Chapter 9: Deploying and Monitoring Pipelines in Production
The journey of data is fast approaching its destination. In the previous chapter, we learned about data aggregation and how to build the gold layer of a lakehouse. Data in the gold layer is the representation of summary data, which keeps changing every time new data gets ingested. Therefore, it is a cyclical process. Now that we have built and tested all of the necessary pieces of the Electroniz lakehouse, we need to start thinking about deploying the pipeline in production. This is so that it can serve the needs of the end users in the most timely and durable fashion.
In this chapter, we will cover the following topics:
- The deployment strategy
- Developing the master pipeline
- Testing ...
Get Data Engineering with Apache Spark, Delta Lake, and Lakehouse now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
