Learn how organizations manage their sources of metadata to define where their data comes from and what its meaning is.
Text from the Video:
If you’ve ever received a package in the mail then you’re probably familiar with shipment tracking. We track our shipments because we want to know where our stuff is and where it came from. That way if something goes wrong (like say your package comes tangled in some bagpipes) you can trace back to the possible source of the problem. Well the same is true for data. Every day organizations move millions of records into different systems, data warehouses, reporting systems, staging areas etc. And organizations need to know where this data is going.
If you are unable to see the videos because of your corporate network restrictions, please fill out the “Talk with a Specialist” form below and indicate the issues you are having. We will work out a solution.
For example, lets imagine that you’re looking at a report and the graph is showing a new division that you know doesn’t exist. So you ask yourself, where did this data come from? For many organizations this question can be a mess to answer. This is because the data movement process can be a black box to business analysts. So this can kick off an expensive analysis effort to document all the hops the data went through to land in that report. Now, if this need only happened say once a year, maybe organizations could live with that analysis effort. But in a data rich environment with lots of data consumers these question is going to be asked regularly. And its not just the business users that will be asking these questions, but IT team as well. This is because IT teams need to understand the impact of the changes they make to the source systems on the downstream reports. A change to a single table could impact hundreds of reports that eventually use that table.
And this is where I introduce you to something called Metadata Management. Pause for a second: Notice I said METADATA Management and not MASTER DATA Management. It just so turns out that within all these hops, each one of these systems keeps a log of the data that’s being handled. These logs are metadata, or information about the data. By tying together the different metadata sources you can begin to get a picture of where the data is moving and how it is changing. Now if you had to tie this all together by hand you would have a massive hairball of complexity to piece together. But luckily a number of the leading vendors in data integration have prebuilt some of this for you. They’ve done this by packaging connectors to the metadata which is stored in industry leading ETL tools, BI tools, databases, data modeling tools, and other systems. So as long as you have a fairly industry standard tool in those categories, there’s a good chance you’ll be able to draw metadata out of those tools without a massive effort. If you have an obscure ETL tool or BI tool then you’re out of luck. You’re left with the options of either hand integrating the metadata into a metadata management tool or living without visibility into that step, and sometimes you can live without that visibility. However, there is one source of metadata that you can’t or shouldn’t live without, and that is your ETL tool. For a refresher on what an ETL tool does check out my ETL video. The entire data movement process is driven by your ETL tool so you’ve got to ensure that you have adequate visibility in that category.
The end result of an implemented Metadata Management tool is a full data tracing map. Which gives you insight coming from either angle. You can look at it from a developers perspective and ask which reports use this table I’m about to edit? Or you can look at it from a business users perspective and ask how did this data end up in my report? We call these Impact Analysis and Data Linage.
Now in addition to Impact Analysis and Data Lineage, Metadata Management tools offer another huge value add, which is called a Business Glossary. Have you ever been in a report and seen a column title like “Shipment Date” and wondered what that means? That term could mean a number of different things, wouldn’t it be nice if there was a central location for storing business descriptions for such data? That is exactly what Business Glossary intended to do. The leading Metadata Management vendors provide this Business Glossary as an interface to enable business users to create descriptions for the data. These descriptions can be collaborated on and run through an approval process. Additionally many of these vendors provide the ability to combine terms into calculations which are linked to each other. This is another area where it’s important to understand that there is no pixie dust fiery that is going to provide these business descriptions. It requires an engaged Data Governance organization to ensure those tasks are taken care of by the business user community. You can get more information on the basics of a Data Governance team by clicking on this video.
Intricity has an entire strategic offering around establishing a Metadata Management practice. This includes engaging with a Data Governance team and helping the organization build connectors to non-standard hops of metadata. I recommend you reach out to Intricity and talk with one of our specialists. We can help you untangle that data movement mess into a self-documenting map which business and IT can collaborate in.
