Data Observability for Data Engineering

5

Defining Rules on Indicators

In the previous chapters, we saw how you could collect events synchronously in your data applications. We also discussed what contextual information you need in order to draw the big picture of what’s happening inside the applications.

Now that you have a lot of contextual information, it is high time to turn it into actionable insights. The metrics you collect during the pipeline execution need to reassure all the stakeholders about the proper execution of the data applications. All the observers of the pipeline need to be informed about how the data pipeline is behaving.

To maintain the trust of data producers and data consumers, we will introduce the concept of expectations, which will define what the engineer ...