Data Pipelines with Apache Airflow

5 Defining dependencies between tasks

This chapter covers

• Examining how to define task dependencies in an Airflow DAG
• Explaining how to implement joins using trigger rules
• Showing how to make tasks conditional on certain conditions
• Giving a basic idea of how trigger rules affect the execution of your tasks
• Demonstrating how to use XComs to share state between tasks
• Examining how Airflow 2’s Taskflow API can help simplify Python-heavy DAGs

In previous chapters, we saw how to build a basic DAG and define simple dependencies between tasks. In this chapter, we will further explore exactly how task dependencies are defined in Airflow and how these capabilities can be used to implement more complex patterns, including conditional tasks, branches, ...