12 Operating Airflow in production

This chapter covers

  • Dissecting the Airflow scheduler
  • Configuring Airflow to scale horizontally using different executors
  • Monitoring the status and performance of Airflow visually
  • Sending out alerts in case of task failures

In most of the previous chapters, we focused on various parts of Airflow from a programmer’s perspective. In this chapter, we aim to explore Airflow from an operations perspective. A general understanding of concepts such as (distributed) software architecture, logging, monitoring, and alerting is assumed. However, no specific technology is required.

Configuring Airflow

Throughout this chapter, we often refer to the Airflow configuration. Configuration in Airflow is interpreted in this ...

Get Data Pipelines with Apache Airflow now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.