Skip to Main Content
Fundamentals of Data Observability
book

Fundamentals of Data Observability

by Andy Petrella
August 2023
Beginner to intermediate content levelBeginner to intermediate
264 pages
7h 15m
English
O'Reilly Media, Inc.
Book available
Content preview from Fundamentals of Data Observability

Chapter 5. Automate the Generation of Data Observations

With the challenges of low-level logging introduced in the previous chapter, our focus shifts to exploring alternative approaches to enhance data observability and streamline its adoption, via automation. In this chapter, I will introduce new possibilities and strategies for capturing and analyzing data observations, paving the way for a more comprehensive observability framework.

Abstraction Strategies

The previous chapter explained how to add data observability to applications that deal with data using a low-level API. This approach can become tedious, even for simple processes, leaving the task to the final user—the data engineer or practitioner.

Most of the work, such as the code instructions presented in Chapter 4, can be abstracted into a higher-level framework that reduces the amount of effort involved, as long as you dedicate enough time to defining it. Like other best practices, the pressure to reduce time to market can cause the generation of data observations to be skipped, even when the engineers know that they will deploy something in which they have little or no confidence.

To avoid falling into this trap, we can automate various best practice strategies or techniques—this is a classic optimization practice in other areas such as application observability, where most common observations are generated by default, such as the number of requests for a web service, the memory usage of a computation engine, the ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

Architecting Data and Machine Learning Platforms

Architecting Data and Machine Learning Platforms

Marco Tranquillin, Valliappa Lakshmanan, Firat Tekiner
Fundamentals of Data Engineering

Fundamentals of Data Engineering

Joe Reis, Matt Housley
Fundamentals of Data Engineering

Fundamentals of Data Engineering

Joe Reis, Matt Housley

Publisher Resources

ISBN: 9781098133283Errata PageSupplemental Content