1 Meet Apache Airflow
This chapter covers
- Showing how data pipelines can be represented in workflows as graphs of tasks
- Understanding how Airflow fits into the ecosystem of workflow managers
- Determining if Airflow is a good fit for you
People and companies are continuously becoming more data-driven and are developing data pipelines as part of their daily business. Data volumes involved in these business processes have increased substantially over the years, from megabytes per day to gigabytes per minute. Though handling this data deluge may seem like a considerable challenge, these increasing data volumes can be managed with the appropriate tooling.
This book focuses on Apache Airflow, a batch-oriented framework for building data pipelines. ...
Get Data Pipelines with Apache Airflow now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
