Cost-Effective Data Pipelines

Preface

In my work on data pipelines, the costliest event I’ve seen was due to a bug: a pipeline was transforming data incorrectly for months, and the problem went undetected until our customers noticed the data was wrong.

As is often the case, many issues led to this outcome. The data was highly variable, making data quality monitoring difficult. We had test data, but it was woefully out of date. The only way to test code changes was with a full pipeline run, which was both long running and costly. We knew the data source could change unpredictably, but we didn’t have data validation in the pipeline to detect when changes occurred.

We could have caught this bug with schema-based validation, which you’ll learn about in this book. Instead, we spent a significant chunk of our annual cloud bill recomputing the bad data. As if that wasn’t bad enough, it cost us the trust of our customers as well, to the point where the validity of the project was questioned. A multimillion-dollar contract supporting a dozen jobs to deliver a service that assisted nearly 100 million people was on the line. Errors of this scale are something every data engineer with significant responsibilities will be exposed to in their careers.

When trying to rein in costs for cloud data pipelines, you’re often confronted with trade-offs. Hit performance requirements, but reduce wasted compute cycles. Use observability for debugging, reducing costs, and evolving designs, but don’t overspend on monitoring and logging. ...