7 Communicating with external systems

This chapter covers

  • Working with Airflow operators performing actions on systems outside Airflow
  • Applying operators specific to external systems
  • Implementing operators in Airflow doing A-to-B operations
  • Testing tasks connecting to external systems

In all previous chapters, we’ve focused on various aspects of writing Airflow code, mostly demonstrated with examples using generic operators such as the BashOperator and PythonOperator. While these operators can run arbitrary code and thus could run any workload, the Airflow project also holds other operators for more specific use cases, for example, running a query on a Postgres database. These operators have one and only one specific use case, such as running ...

Get Data Pipelines with Apache Airflow now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.