The Fundamentals of Telemetry Pipelines

Chapter 2. The Domain Language of Telemetry Pipelines

There is a tide in the affairs of men

Which, taken at the flood, leads on to fortune;

Omitted, all the voyage of their life

Is bound in shallows and in miseries.

On such a full sea are we now afloat.

And we must take the current when it serves,

Or lose our ventures.

William Shakespeare, Julius Caesar

In Chapter 1, you saw how telemetry data has become a flood. A flood where the properties of your telemetry data need to be considered so that you can get the most value out of all the data that is available. In this chapter, you’re going to explore the foundational concepts of telemetry pipelines so that you can begin to do exactly that.

The Building Blocks of Telemetry Pipelines

Telemetry pipelines are made up of five concepts: sources, streams, processors, destinations, and, of course, the pipeline itself. Sources are your faucets for the flood of telemetry data being emitted from your systems. Each source produces a stream of structured telemetry data. Processors are the workhorses that do what it takes to manipulate those streams of telemetry data you’ve tapped into. Destinations are your channels to bring this refined telemetry data to downstream tools and other systems. Finally, the pipeline packages all these sources, streams, processors, and destinations into a unit you can work with and manage. Let’s take a closer look.

Starting at the Source

Most data pipelines start by answering the question, “Where are we going ...