The Fundamentals of Telemetry Pipelines

Chapter 6. Conclusion

Railroad iron is a magician’s rod in its power to evoke the sleeping energies of land and water.

Ralph Waldo Emerson

On the face of it, telemetry pipelines don’t sound like they will change your world, but like the railroads in Ralph Waldo Emerson’s quote, there’s an enormous amount of power in their simplicity. From just the four building blocks of sources, streams, processors, and destinations, you can build pipelines that can help you solve problems more quickly, save your organization money, or even avoid huge compliance fines. And those are just the cases we’ve had time to cover here.

When you invest the time necessary to look at and think differently about your telemetry data, you can start to treat it as the valuable resource it really is. No longer is it merely a peripheral side effect; your telemetry data—metrics, traces, logs, and other events like alerts—is transformed into something visualizable, manageable, and, most importantly, valuable.

You get to streamline what gets through to your various live observability downstream tools, hedging your bets by storing anything you don’t think you’ll need today in a cheap backup location. Then the world opens up to deliver better business insights, improve your security posture, accelerate incident resolution times, improve customer experience, and, as described in the previous chapter, even ensure that you meet all required data compliance regulations. And all of that is likely just scratching the ...