7
Optimizing Data Pipelines
The importance of data in companies has significantly increased the investments in data platforms by companies. Over time, this has increased companies’ priority of being aware of what their data pipelines do and how they do it and therefore monitoring not only the quality of the outcomes but also the state of health of the pipelines. At the same time, they are also monitoring the usage of the resources and tracking the associated costs.
In this chapter, we will understand how data observability offers us a way to make the governance of our data pipelines scalable and sustainable. First, we will focus on understanding the key data pipelines, their main components, and the types of data pipelines, as well as their ...
Get Data Observability for Data Engineering now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
