Automating Data Quality Monitoring at Scale

Chapter 2. Data Quality Monitoring Strategies and the Role of Automation

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 2nd chapter of the final book.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at [e-mail].

There are many different ways you can approach data quality monitoring. Before evaluating the options, it helps to think about what success looks like. In this chapter, we’ll define the requirements for success. Then we’ll walk through the traditional strategies—manual checks, rule-based testing, and metrics monitoring—and see how they measure up.

Next, we’ll explore the idea of automating data quality monitoring. We’ll explain how unsupervised machine learning can help us satisfy some missing aspects of our success criteria, scaling monitoring to large amounts of data while reducing alert fatigue.

We’ll wrap up by introducing the data quality monitoring strategy we advocate for in this book: a four-pillar approach combining data observability, rule-based testing, metrics monitoring, and unsupervised machine learning. As we’ll show, this approach has many advantages. It allows subject matter experts to enforce essential ...