Site Reliability Engineering, 2nd Edition

Chapter 9. Incident Management and On-call

By Dan Dennison, David Huska, Garrett Plasky, Jennifer Mace, and Michelle Brush

with Garrett Holthaus

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 10th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

Complex systems fail. This is an unavoidable reality of software engineering at scale. When they do, organized incident management and on-call practices are the difference between a system that is quickly and smoothly restored to full operation and one which remains unhealthy longer than necessary.

In this chapter, we provide a practical guide to navigating broken systems. We cover foundational concepts around how to set up sustainable on-call rotations and organize your incident response to effectively triage, mitigate, and resolve outages. We’ll discuss our model for incident response at Google, identify common organizational pitfalls, and offer technical and cultural solutions to them.

We also examine how AI and ML fit into incident response, exploring their current and expected capabilities and risks in this domain, and we’ll specifically address ...