May 2007
Intermediate to advanced
432 pages
11h 43m
English
Content preview from Expert SQL Server™ 2005 Integration Services
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
3.3. Tracking Data Lineage Identifiers
Data lineage is a part of the metadata tracking in an ETL process associated with mapping data back to the source data points. But before considering an approach to data lineage tracking, to better understand data lineage requires understanding metadata in general.
Metadata is the data about the data. For ETL, it's the integrated tracking mechanism for the process and the core data. Metadata is very important for ETL and data warehousing—not just for administrators to understand what data got loaded, but also for system developers to validate data sources, for data administrators to find out what happened when failures occur, and for users to be confident in the data they are looking at. In fact, the latter may be the most important, because if the users do not trust the data, then their adoption of the solution will be slow going. Tracking data lineage is directly related to knowing where each data point came from, not just what system or table or file, but knowing which source row(s) they came from.
As with other requirements, data lineage can be handled several ways in SSIS. The most common way is to add an auto-incrementing number to the data flow that maps data points back to their sources.
If you are staging raw data, this can be leveraged by sending this lineage number to the staged data and also to the destination tables. With raw staging tables, instead of having to track back to a source system, which may be volatile, you are able ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.