Chapter 5. Automate the Generation of Data Observations

With the challenges of low-level logging introduced in the previous chapter, our focus shifts to exploring alternative approaches to enhance data observability and streamline its adoption, via automation. In this chapter, I will introduce new possibilities and strategies for capturing and analyzing data observations, paving the way for a more comprehensive observability framework.

Abstraction Strategies

The previous chapter explained how to add data observability to applications that deal with data using a low-level API. This approach can become tedious, even for simple processes, leaving the task to the final user—the data engineer or practitioner.

Most of the work, such as the code instructions presented in Chapter 4, can be abstracted into a higher-level framework that reduces the amount of effort involved, as long as you dedicate enough time to defining it. Like other best practices, the pressure to reduce time to market can cause the generation of data observations to be skipped, even when the engineers know that they will deploy something in which they have little or no confidence.

To avoid falling into this trap, we can automate various best practice strategies or techniques—this is a classic optimization practice in other areas such as application observability, where most common observations are generated by default, such as the number of requests for a web service, the memory usage of a computation engine, the ...

Get Fundamentals of Data Observability now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.