97 Things Every Data Engineer Should Know

Chapter 20. Data Quality for Data Engineers

Katharine Jarmul

If you manage and deploy data pipelines, how do you ensure they are working? Do you test that data is going through? Do you monitor uptime? Do you even have tests? If so, what exactly do you test?

Data pipelines aren’t too dissimilar to other pipelines in our world (gas, oil, water). They need to be engineered; they have a defined start and end point. They need to be tested for leaks and regularly monitored. But unlike most data pipelines, these “real-world” pipelines also test the quality of what they carry. Regularly.

When was the last time you tested your data pipeline for data quality? When was the last time you validated the schema of the incoming or transformed data, or tested for the appropriate ranges of values (i.e., “common sense” testing)? How do you ensure that low-quality data is either flagged or managed in a meaningful way?

More than ever before—given the growth and use of large-scale data pipelines—data validation, testing, and quality checks are critical to business needs. It doesn’t necessarily matter that we collect 1TB of data a day if that data is essentially useless for tasks like data science, machine learning, or business intelligence because of poor quality control.

We need data engineers to operate like other pipeline engineers—to be concerned about and focused on the quality of what’s running ...