February 2021
Beginner to intermediate
274 pages
5h
English
Content preview from Data Pipelines Pocket Reference
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
Chapter 7. Orchestrating Pipelines
Previous chapters have described the building blocks of data pipelines, including data ingestion, data transformation, and the steps in a machine learning pipeline. This chapter covers how to “orchestrate,” or tie together, those blocks or steps.
Orchestration ensures that the steps in a pipeline are run in the correct order and that dependencies between steps are managed properly.
When I introduced the challenge of orchestrating pipelines in Chapter 2, I also introduced the concept of workflow orchestration platforms (also referred to as workflow management systems (WMSs), orchestration platforms, or orchestration frameworks). In this chapter, I will highlight Apache Airflow, which is one of the most popular such frameworks. Though the bulk of the chapter is dedicated to examples in Airflow, the concepts are transferable to other frameworks as well. In fact, I note some alternatives to Airflow later in the chapter.
Finally, the later sections of this chapter discuss some more advanced concepts in pipeline orchestration, including coordinating multiple pipelines on your data infrastructure.
Directed Acyclic Graphs
Though I introduced DAGs in Chapter 2, it’s worth repeating what they are. This chapter talks about how they are designed and implemented in Apache Airflow to orchestrate tasks in a data pipeline.
Pipeline steps (tasks) are always directed, meaning they start with a task or multiple tasks and end with a specific task or tasks. This ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.