Cost-Effective Data Pipelines

Chapter 2. Responding to Changes in Demand by Scaling Compute

When I’m not writing a book about data pipelines, I like to spend my free time exploring the woods. A big part of planning these outings involves figuring out what clothing to take. The weather forecast, how strenuous the hike is, and my pace all play a factor in this decision.

Dressing in layers is the time-honored approach to this conundrum. If I’m walking briskly up a hill, I’ll remove some layers to avoid overheating. When I stop at the top of the hill to admire the view, I’ll put them back on to stay warm. Being able to add and remove layers keeps me comfortable and safe in the backcountry.

Much like I can change layers to accommodate different conditions while hiking, you can customize the amount of resources for a pipeline to accommodate different workloads. Think of this as dynamically right-sizing compute, an iterative process that involves monitoring and tuning to dial things in.

Note

In Chapter 1 you saw the definition of vertical scaling, which changes the capacity of a resource. This chapter focuses on horizontal scaling, which changes the number of resources. Adding more resources is scaling out and removing resources is scaling in, as described in “Design for Scaling in the Microsoft Azure Well-Architected Framework”.

Scaling workloads is another realm of cost–performance trade-offs. You want to scale in resources to save cost, but not at the expense of pipeline reliability and performance. You want ...