Site Reliability Engineering, 2nd Edition

Chapter 12. Safety Engineering for SRE with STPA and CAST

By Garrett Holthaus, Ruben Barroso, and Tim Falzone with Theo Klein

Traditional incident response focuses on fixing what broke. But what happens when a major outage occurs without a single component failing? This chapter explores how Google is adopting systems thinking and safety engineering to provide a proactive approach to reliability, helping to anticipate and prevent incidents before they occur.

On one eventful day in 2019, if you went to Google Maps to search for places in Japan, you likely couldn’t find them. As with other outages, SRE was paged and collaborated with product developers to mitigate the incident. We wrote a postmortem once the issue was resolved and Maps in Japan was back to normal. As postmortems go, this was a very good one. It was thoroughly researched and provided a clear and complete explanation of the underlying problems and the sequence of events that led up to the incident. The postmortem included some discussion of how human decision-making contributed to the incident, without blaming anyone. It identified and prioritized action items, and multiple engineering teams—including SRE—started projects to prevent this kind of problem from occurring again.

While we’re never happy when something like this happens, so far this sounds like a successful incident response and learning outcome for the organization. Still, a few people had lingering doubts. Despite the thorough postmortem, the SRE lead ...