The Future of Observability with OpenTelemetry

Chapter 8. How to Roll Out OpenTelemetry Across Your Organization

Rolling out a new telemetry system can be a complex endeavor. It requires buy-in from across the engineering organization, and usually cannot be deployed in a single step. Don’t underestimate the problems this may create!

In a large organization, there are usually many service teams that own different parts of the system. Often, each team will need to put in some amount of effort to get the services they manage fully instrumented. And those teams have their own backlog of work, which of course they want to prioritize.

An unfortunate result is that observability initiatives can run out of steam before they can begin to provide value and prove their worth. But with careful planning and coordination, this can be avoided.

The Primary Goal

When rolling out OpenTelemetry, it’s important to remember that any observability system based on distributed tracing will require every service participating in a transaction to be instrumented to provide maximum value. If only some of the services are instrumented, the traces will be fragmented into small, disconnected pieces.

The results of this kind of scattershot instrumentation are unimpressive. This situation—inconsistent instrumentation and broken traces—is the main thing you want to avoid. If the traces are disconnected, operators will still need to stitch everything together in their heads to get a picture of their system. Worse, automated analysis tools will have very ...