Rebuilding Reliable Data Pipelines Through Modern Tools

Chapter 1. Introduction

Back in my 20s, my wife and I started running in an attempt to fight our ever-slowing metabolism as we aged. We had never been very athletic growing up, which comes with the lifestyle of being computer and video game nerds.

We encountered many issues as we progressed, like injury, consistency, and running out of breath. We fumbled along making small gains and wins along the way, but there was a point when we decided to ask for external help to see if there was more to learn.

We began reading books, running with other people, and running in races. From these efforts we gained perspective on a number of areas that we didn’t even know we should have been thinking about. The perspectives allowed us to understand and interpret the pains and feelings we were experiencing while we ran. This input became our internal monitoring and alerting system.

We learned that shin splints were mostly because of old shoes landing wrong when our feet made contact with the ground. We learned to gauge our sugar levels to better inform our eating habits.

The result of understanding how to run and how to interpret the signals led us to quickly accelerate our progress in becoming better runners. Within a year we went from counting the blocks we could run before getting winded to finishing our first marathon.

It is this idea of understanding and signal reading that is core to this book, applied to data processing and data pipelines. The idea is to provide a high- to mid-level introduction ...