Site Reliability Engineering, 2nd Edition

Chapter 9. Observability and Monitoring

By Alex Bramley, Amber Hillhouse, Boris Arnoux, Brent Bryan, Brian Kudzia, David Bayer, Josh Suereth, Jose Andrade, Kyusoon Lee, Leland Jefferis, Marcin Brodziak, Micah Lerner, Omer Habas, Parker Roth, Pereira Braga, Qin Jia, Ralf Wildenhues, Raphael Menderico, Stevan Malesevic, Yael Evron Sabag, and Ying Wang with Carmela Quinito

Monitoring has undergone a significant transformation in recent years. Historically, the primary objective was to meticulously collect and store state data from various components within distributed systems. These components included a wide array of elements such as client applications, frontend interfaces, backend services, storage solutions, data pipelines, hardware infrastructure, and other interconnected systems.

The practice of troubleshooting production issues with traditional monitoring data was overly dependent on:

Institutional knowledge transfer

Crucial insights and operational nuances were communicated primarily through direct interaction, often from experienced on-call engineers to their more junior counterparts. This method, while valuable for direct mentorship, was inherently inefficient for scaling knowledge across larger teams and posed a risk of knowledge loss if key personnel departed.

Manual correlation for hypothesis generation

When an incident occurred, engineers would manually sift through disparate data points to identify potential correlations and formulate hypotheses about the underlying ...