The Site Reliability Workbook

Chapter 4. Monitoring

By Jess Frame, Anthony Lenton, Steven Thurgood, Anton Tolchanov, and Nejc Trdin with Carmela Quinito

Monitoring can include many types of data, including metrics, text logging, structured event logging, distributed tracing, and event introspection. While all of these approaches are useful in their own right, this chapter mostly addresses metrics and structured logging. In our experience, these two data sources are best suited to SRE’s fundamental monitoring needs.

At the most basic level, monitoring allows you to gain visibility into a system, which is a core requirement for judging service health and diagnosing your service when things go wrong. Chapter 6 in the first SRE book provides some basic monitoring definitions and explains that SREs monitor their systems in order to:

• Alert on conditions that require attention.

• Investigate and diagnose those issues.

• Display information about the system visually.

• Gain insight into trends in resource usage or service health for long-term planning.

• Compare the behavior of the system before and after a change, or between two groups in an experiment.

The relative importance of these use cases might lead you to make tradeoffs when selecting or building a monitoring system.

This chapter talks about how Google manages monitoring systems and provides some guidelines for questions that may arise when you’re choosing and running a monitoring system.

Desirable Features of a Monitoring Strategy

When choosing a monitoring ...