Building Modern Data Applications Using Databricks Lakehouse

10

Monitoring Data Pipelines in Production

In the previous chapters, we learned how to build, configure, and deploy data pipelines using the Databricks Data Intelligence Platform. To round off managing data pipelines for the lakehouse, in this final chapter of the book, we’ll dive into the crucial task of monitoring data pipelines in production. We’ll learn how to leverage comprehensive monitoring techniques directly from the Databricks Data Intelligence Platform to track pipeline health, pipeline performance, and data quality, to name a few. We will also implement a few real-world examples through hands-on exercises. Lastly, we’ll look at the best practices for ensuring that your data pipelines run smoothly, enabling timely issue detection ...