Site Reliability Engineering, 2nd Edition

Chapter 3. Safety Engineering for Software: Beyond Reacting to Failure

Garrett Holthaus, Ruben Barroso, and Tim Falzone

Acknowledgements: David Ferguson and Theo Klein

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 13th chapter of the final book.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

Traditional incident response focuses on fixing what broke. But what happens when a major outage occurs without a single component failing? This chapter explores how Google is adopting systems thinking and safety engineering to provide a proactive approach to reliability, helping to anticipate and prevent incidents before they happen.

For one eventful day in 2019, if you came to Google Maps to search for places in Japan, you mostly couldn’t find them. As with other outages, SRE got paged and collaborated with product developers to mitigate the incident. We wrote a postmortem once the issue was resolved and Maps in Japan was back to normal. As postmortems go, this was a very good one. It was thoroughly researched, and had a clear and complete explanation of the underlying problems and the sequence of events that led up to the incident. The postmortem included some discussion on how human ...