97 Things Every Data Engineer Should Know

Chapter 53. Observability for Data Engineers

Barr Moses

As companies become increasingly data driven, the technologies underlying the rich insights data provides have grown more nuanced and complex. Our ability to collect, store, aggregate, and visualize this data has largely kept up with the needs of modern data teams (think domain-oriented data meshes, cloud warehouses, and data-modeling solutions), but the mechanics behind data quality and integrity have lagged.

How Good Data Turns Bad

After speaking with several hundred data-engineering teams, I’ve noticed three primary reasons for good data turning bad:

More and more data sources

Nowadays, companies use anywhere from dozens to hundreds of internal and external data sources to produce analytics and ML models. Any one of these sources can change in unexpected ways and without notice, compromising the data the company uses to make decisions.

Increasingly complex data pipelines

Data pipelines are increasingly complex, with multiple stages of processing and nontrivial dependencies among various data assets. With little visibility into these dependencies, any change made to one dataset can impact the correctness of dependent data assets.

Bigger, more specialized data teams

Companies are increasingly hiring more and more data analysts, scientists, and engineers to build and maintain the data pipelines, analytics, and ML models ...