Data Transformation: The Definitive Guide

Chapter 2. Backfilling and Reprocessing

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 4th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

Change is constant. Business priorities shift, schemas evolve, bugs pop up, and new compliance rules can change your whole pipeline strategy. These changes usually mean you need to look backward and reprocess historical data to match your current reality. This chapter digs into backfilling and reprocessing, taking what most teams treat as a huge pain and turning it into something strategic.

Look at Netflix’s data engineering team. They cut their compute time and costs dramatically by switching to incremental processing with Apache Iceberg and Maestro.¹ This wasn’t because they bought faster hardware or hired more engineers. They just understood the fundamental principles of effective data reprocessing and applied them. These principles, patterns, and practices are what we’re covering here.

Backfilling takes resources and comes with real risks, but you can’t maintain data quality and consistency without it. Gartner’s research shows that poor data quality ...