O'Reilly logo

The Site Reliability Workbook by Stephen Thorne, Kent Kawahara, David K. Rensin, Niall Richard Murphy, Betsy Beyer

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Appendix C. Results of Postmortem Analysis

At Google, we have a standard postmortem template that allows us to consistently capture the incident root cause and trigger, which enables trend analysis. We use this trend analysis to help us target improvements that address systemic root-cause types, such as faulty software interface design or immature change deployment planning. Table C-1 shows the breakdown of our top eight triggers for outages, based on a sample of thousands of postmortems over the last seven years.

Table C-1. Top eight outage triggers, 2010–2017
Binary push 37%
Configuration push 31%
User behavior change 9%
Processing pipeline 6%
Service provider change 5%
Performance decay 5%
Capacity management 5%
Hardware 2%

Table C-2 presents the top five contributing root-cause categories.

Table C-2. Top five root-cause categories for outages
Software 41.35%
Development process failure 20.23%
Complex system behaviors 16.90%
Deployment planning 6.74%
Network failure 2.75%

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required