Site Reliability Engineering, 2nd Edition

Chapter 11. Learning from Incidents

By James Brookbank, Michelle Brush, Sue Lueder, and Tim Falzone with Garrett Holthaus

When the goal is reliable systems operating at scale, outages are more than disruptions to be resolved effectively; they are invaluable learning opportunities. For over a decade, the practices and culture of Site Reliability Engineering at Google have supported continual and iterative learning from our observed and narrowly avoided system failures. This fundamental aspect of SRE emphasizes blameless postmortems, psychological safety, collaboration, and executive support in order to understand what happened and improve the overall system, obtaining maximum benefit from each outage.

This chapter covers these enduring core tenets of incident analysis and delves into how our understanding and application of them have evolved over the past 10 years. You’ll discover how we’ve refined our practices, leveraged advanced tooling, and deepened our appreciation for the human element, allowing us to learn from incidents to build more resilient and robust services.

Principles and Practices

The original SRE Book and Chapter 10 of The Site Reliability Workbook covered key principles and practices for learning from incidents through the deeply ingrained blameless postmortem culture at Google. Ten years on, the need to learn from failure remains central to building and operating resilient systems. This section walks through those foundational lessons on incident analysis and ...