Observability Engineering, 2nd Edition

Chapter 31. Instrumentation for Observability Teams

This chapter was cowritten by Hazel Weakly, Fellow at the Nivenly Foundation, and Austin Parker, Director of AI Strategy at Honeycomb.

Ralph Waldo Emerson once wrote, “language is fossil poetry.”¹ Social scientists and philosophers have long observed that language itself both shapes (and is shaped by) society. Wherever two or more people gather, a language is sure to be either developed or refined. Culture itself is the crucible of language—the way we interact and relate to one another.

An observability engineering team grappling with an instrumentation strategy for their system faces similar burdens. Code ages like the value of a new car driven off the lot: it depreciates the moment it’s written. While software (as a whole) may provide value, code itself contains a burden of maintenance and care that requires careful planning, strategy, and gardening in order to avoid overwhelming operators as their system scales. If that describes the kind of problems that keep you up at night, then you’re in the right place.

Observability teams may take different forms depending on the size, scale, and remit of an organization. In smaller orgs, “observability team” may be a role that’s foisted upon a site reliability engineer. Larger organizations may have multiple teams, each responsible for a certain set of logical systems. Your bosses may be in information technology, or they might be in software engineering. Your job may look more like ...