97 Things Every Data Engineer Should Know

Chapter 74. The Importance of Data Lineage

Julien Le Dem

As a data engineer, you become a sort of collector of datasets coming from various sources. The challenge is that datasets don’t just stay pinned on a board behind glass. They have to be maintained and updated in a timely manner. They have to be transformed and adapted to various use cases. They change form over time; all the layers built upon them have to be updated.

As data pipelines pile up, complexity increases dramatically, and it becomes harder to keep things updated reliably in a timely manner. Observing data lineage in all the layers of transformation—from ingestion to machine learning, business intelligence, and data processing in general—provides a critical source of visibility. With this information, the engineer on call can understand what’s happening and resolve problems quickly when a crisis happens.

Lineage provides the understanding of how a dataset was derived from another one. Operational lineage goes one step beyond by tracing how and when that transformation happened. It captures information such as:

• The version of the input that was consumed

• The subset of the data that was read

• The version of the code doing the transformation

• The output’s definition and how each column was derived from the input

• The time it took to complete and whether it was successful

• The version of the output that was produced