Cost-Effective Data Pipelines

Chapter 12. Essential Takeaways

As you might imagine, my initial research for this book involved reading a lot of material about the cost of cloud services. From shell-shocked graduate students grappling with unexpected bills to large companies feeling trapped with substantial, expensive cloud deployments, it was clear that developing data pipelines in the cloud can be daunting.

It reminds me of learning to ride waves on a bodyboard when I was a kid. Similar to surfing, riding waves on a bodyboard requires that you develop a sense of when to start paddling to catch a wave at the right time. If you don’t time it right, you can miss the wave or get dunked when the wave crashes on top of you.

I got dunked a lot in the beginning, ending up with a nose full of saltwater, but gradually I got better. I developed a sense of how the strength of the undertow related to the incoming wave. I figured out how to angle the board to get a better ride. Sometimes I still got dunked.

This was how I felt when I started working in the cloud, a few years after I began working on data pipelines. The steep learning curve was no joke. A big motivation for writing this book was wishing I had something like it at the time. Data pipelines and cloud development are two big topics on their own, let alone together. Add to it the desire to cut costs and you’ve got quite a lot to digest.

In reflecting on the last 240ish pages, I want to wrap things up by distilling this volume down to what I consider to be the ...