Observability Engineering, 2nd Edition

Chapter 9. Observability-Driven Development

In Part II, we introduced instrumentation: what to emit and why. In the previous chapter, we focused on analysis: how telemetry becomes evidence that you can use to understand system behavior. This chapter closes the loop between those two actions.

Observability is often framed as something you use after a service is in production. That framing is incomplete. Observability should also be an early part of your software development practices. Telemetry should evolve alongside your code, and engineers should think about instrumentation any time a change could introduce new failure modes or shift performance in ways that matter to users.

That’s the core idea behind observability-driven development; a practice similar to test-driven development (TDD), but covering different concerns. When observability drives code development, you’ll ship new features together with the instrumentation needed to measure impacts and capture the context you’ll need to verify their performance, compare them to previous versions, and debug them quickly when reality disagrees with your assumptions.

We’ll start by reviewing TDD and its limits when applied to production environments. We’ll look at how observability complements TDD by enabling rapid triage, systematic investigation in distributed systems, and shorter feedback loops between features and customer experiences. Lastly, we’ll cover how this approach supports progressive delivery and how generative AI can ...