Site Reliability Engineering, 2nd Edition

Chapter 10. Incident Management and On-Call

By Dan Dennison, David Huska, Garrett Plasky, Jennifer Mace, and Michelle Brush with Garrett Holthaus

Complex systems fail. This is an unavoidable reality of software engineering at scale. When they do, organized incident management and on-call practices are the difference between a system that is quickly and smoothly restored to full operation and one that remains unhealthy or unavailable for longer than necessary.

In this chapter, we provide a practical guide to navigating broken systems. It covers foundational concepts around how to set up sustainable on-call rotations and organize your incident response to effectively triage, mitigate, and resolve outages. We’ll discuss our model for incident response at Google, identify common organizational pitfalls, and offer technical and cultural solutions to them.

We’ll also specifically address the human element: the specific, learnable skills required for incident response and how to ensure your team is able to respond effectively when everything seems to be going wrong.

Finally, we’ll examine how AI and ML fit into incident response, exploring their current and expected capabilities and risks in this domain. The chapter wraps up with a case study illustrating how Etsy applies the principles discussed in this chapter to its incident management practices.

Incidents, Outages, and Bugs

Let’s start by defining the term incident and how it relates to outages and software bugs. An incident is ...