Chapter 4: Understanding Data Pipelines
In the previous chapter, we discussed the various services that can be used to build a data lake in Microsoft Azure. We will now focus on how to kick-start the process of building a data lake using data pipelines.
As a data engineer, the process of laying out a data pipeline should be extremely well planned. This chapter will educate data engineers regarding the various phases of a data pipeline creation, with a list of recommended actions and tasks that need to be accounted for during each phase. Better planning results in better execution.
In this chapter, we will cover the following topics:
- Exploring data pipelines
- Process of creating a data pipeline
- Running a data pipeline
- Sample lakehouse project ...
Get Data Engineering with Apache Spark, Delta Lake, and Lakehouse now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
