The Fundamentals of Telemetry Pipelines

Chapter 1. The Need for Telemetry Pipelines

Water, water, everywhere, nor any drop to drink.

Samuel Taylor Coleridge, The Rime of the Ancient Mariner

Volume. The problem wasn’t originally volume.

In the early days of managing, monitoring, and operating systems, when things were going well, a system was relatively silent. Your system was a black box that you hoped was working or, if you were lucky, emitted a few signals to give you a sense that things were, indeed, all OK.

In the days of fixed, inflexible systems, fragile and fixed management dashboards were de rigueur. As you composed a system from its individual parts, you, ideally in parallel, decided on the metrics that might be important and looked to expose those for the sole purpose of producing fixed operational management dashboards.

The goal was simple: provide just enough information and control to help the poor souls woken up in the depths of the night to complete their blurry-eyed, urgent, and critical tasks to set the system right, to keep it working, and to keep the lights on.

And then our systems stopped being fixed, resolute, static. Our systems became elastic, rapidly changeable, rapidly scaling and evolving, and the volume of system telemetry signals grew in tandem. With increasing volumes of telemetry data came an increase in opportunity to use that data effectively. This is, in a nutshell, what telemetry pipelines do: help you manage, manipulate, make sense of, and extract value from this new flood of telemetry ...