12 Operating Airflow in production
This chapter covers
- Dissecting the Airflow scheduler
- Configuring Airflow to scale horizontally using different executors
- Monitoring the status and performance of Airflow visually
- Sending out alerts in case of task failures
In most of the previous chapters, we focused on various parts of Airflow from a programmer’s perspective. In this chapter, we aim to explore Airflow from an operations perspective. A general understanding of concepts such as (distributed) software architecture, logging, monitoring, and alerting is assumed. However, no specific technology is required.
Get Data Pipelines with Apache Airflow now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
