October 2023
Beginner
1h 43m
English
Closed Captioning available in German, English, Spanish, French, Italian, Japanese
Overview
Apache Airflow Fundamentals
Course Overview
n this hands-on learning experience, you will gain expertise in building, monitoring, and maintaining data pipelines for workflow orchestration.
This course is ideal for data engineers, analysts, and developers who want to programmatically author, schedule, and monitor complex workflows. We'll guide you through Airflow's key concepts, equipping you with real-world skills to automate and orchestrate data pipelines.
Get started by installing Airflow using one of the recommended methods like Docker, Kubernetes, or PyPI. We'll ensure you have a working environment to start developing pipelines. Leverage Airflow's user-friendly UI to monitor workflow executions, gain insights, and optimize your pipelines.
Master fundamental concepts like directed acyclic graphs, operators, and task dependencies. Learn how to define workflows as code using Python and the rich Airflow API. Develop skills in handling errors, retries, and monitoring pipeline health.
Build upon your foundation by scheduling workflows, setting dependencies, and executing production-grade pipelines. We'll explore real-world scenarios like data manipulation and persistence from remote sources. Airflow's plugins and extensibility will allow you to enhance pipelines with custom functionality.
Our comprehensive curriculum includes hands-on labs, interactive exercises, and projects focused on real-world data engineering workflows. You'll gain expertise across the entire pipeline development lifecycle - from authoring to monitoring, maintenance, and beyond.
Whether you're a beginner or an experienced engineer, this course provides the complete guide to wield the full power of Apache Airflow.
Learning Objectives
- Gain expertise in Airflow concepts like DAGs, tasks, operators, and dependencies to build robust data pipelines.
- Master Airflow installation methods like Docker and PyPI for development and production environments.
- Learn to leverage Airflow's UI to monitor workflow executions, analyze failures, and optimize pipelines.
- Develop skills in authoring workflows as code with Python and utilizing Airflow's extensive APIs.
- Apply real-world skills through hands-on labs focused on data ingestion, transformation, and ML model deployment.
- Gain end-to-end experience across pipeline authoring, scheduling, monitoring, alerting, and maintenance.
These compelling learning objectives will guide you through the course, helping you acquire essential skills and knowledge to become confident in working with Apache Airflow
Course Content
This course has an example respository with a lab challenge you can use to apply everything you've learned. The short videos are focused on small examples at a time and contains hands-on demonstrations. The course is designed to be taken in order, but you can jump to any lesson you want to learn more about.
Complete your learning with the practice lab where you will build a data pipeline using Apache Airflow to extract census data, transform it, and load it into a database based on certain conditions.
About your instructor
Alfredo Deza has over a decade of experience as a Software Engineer doing DevOps, automation, and scalable system architecture. Before getting into technology he participated in the 2004 Olympic Games and was the first-ever World Champion in High Jump representing Peru. He currently works in Developer Relations at Microsoft and is an Adjunct Professor at Duke University teaching Machine Learning, Cloud Computing, Data Engineering, Python, and Rust. With Alfredo's guidance, you will gain the knowledge and skills to build robust data pipelines with Apache Airflow.
Resources
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Watch now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.