Building ETL Pipelines with Python

Part 4:Automating and Scaling ETL Pipelines

In this last section of the book, we will start by covering how to create a robust ETL pipeline in AWS and apply scaling and orchestration strategies. We will introduce the fundamental testing strategies for ETL pipelines and how to both mitigate and quickly handle pipeline failures in a production environment. We will go on to discuss some of the industry’s best practices for creating ETL pipelines in production, as well as identify some of the common pitfalls that engineers should avoid while building enterprise-grade data pipelines. Finally, we will close out this book with three practical case studies of how the tools we’ve introduced throughout this book can be applied to real-life data problems. ...