10 Running tasks in containers
This chapter covers
- Identifying some challenges involved in managing Airflow deployments
- Examining how containerized approaches can help simplify Airflow deployments
- Running containerized tasks in Airflow on Docker
- Establishing a high-level overview of workflows in developing containerized DAGs
In previous chapters, we implemented several DAGs using different Airflow operators, each specialized to perform a specific type of task. In this chapter, we touch on some of the drawbacks of using many different operators, especially with an eye on creating Airflow DAGs that are easy to build, deploy, and maintain. In light of these issues, we look at how we can use Airflow to run tasks in containers using Docker and ...
Get Data Pipelines with Apache Airflow now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
