Cost-Effective Data Pipelines

Appendix. Preparing a Cloud Budget

Throughout this book, you’ve learned how to design and develop data pipelines in a cost-effective way. You’ve evaluated different options of how to allocate compute and storage, making decisions that are appropriate for your current and near-term data pipeline operational goals. You’ve put design strategies in place to reduce the chance of data corruption and recomputation expenses, and you’ve employed development strategies to minimize cloud service costs. On top of this, you have monitoring in place so that you can see how the pipeline performs with the design choices and resource allocations you’ve made.

Going through this process not only helps you put cost-effective designs in place, it also gives you valuable information about where and why you are spending on cloud resources. Even if your day to day doesn’t involve giving budget presentations, communicating how you are saving costs will help propel your career. Engineers who understand cost trade-offs are valuable partners to those who are primarily interested in company financials. You have the ability to act on requests to save costs as well as inform others on what the trade-offs are for doing so.

In this Appendix, you’ll learn how to take this information and create a basic budget for cloud spend, using historical billing data, estimated costs, and pipeline workload expectations.

This is just the tip of the iceberg for budgeting and forecasting, but it will help you leverage what you’ve ...