Fundamentals of Data Observability

Chapter 5. Automate the Generation of Data Observations

With the challenges of low-level logging introduced in the previous chapter, our focus shifts to exploring alternative approaches to enhance data observability and streamline its adoption, via automation. In this chapter, I will introduce new possibilities and strategies for capturing and analyzing data observations, paving the way for a more comprehensive observability framework.

Abstraction Strategies

The previous chapter explained how to add data observability to applications that deal with data using a low-level API. This approach can become tedious, even for simple processes, leaving the task to the final user—the data engineer or practitioner.

Most of the work, such as the code instructions presented in Chapter 4, can be abstracted into a higher-level framework that reduces the amount of effort involved, as long as you dedicate enough time to defining it. Like other best practices, the pressure to reduce time to market can cause the generation of data observations to be skipped, even when the engineers know that they will deploy something in which they have little or no confidence.

To avoid falling into this trap, we can automate various best practice strategies or techniques—this is a classic optimization practice in other areas such as application observability, where most common observations are generated by default, such as the number of requests for a web service, the memory usage of a computation engine, the ...