Site Reliability Engineering, 2nd Edition

Chapter 10. Learning From Incidents

By James Brookbank, Michelle Brush, Sue Lueder, and Tim Falzone

with Garrett Holthaus

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 11th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

When the goal is reliable systems operating at scale, outages are more than disruptions to be resolved effectively; they are invaluable learning opportunities. For over a decade, the practices and culture of Site Reliability Engineering at Google has supported continual and iterative learning from our observed and narrowly avoided system failures. This fundamental aspect of SRE emphasizes blameless postmortems, psychological safety, collaboration, and executive support in order to understand what happened and improve the overall system, obtaining maximum benefit from each outage.

This chapter covers these enduring core tenets of incident analysis and delves into how our understanding and application has evolved over the past ten years. You’ll discover how we’ve refined our practices, leveraged advanced tooling, and deepened our appreciation for the human element allowing ...