Chapter 51. Incident Analysis and Chaos Engineering: Complementary Practices
Ryan Frantz
Learning from failure is extremely useful in a world driven by demands for high reliability.1 Our approach to learning, and which failures we should focus on, may not be clear. Two common avenues to learning are incident analysis and chaos engineering.
Incident Analysis
Well-done incident analysis takes time. Constraining ourselves to a few hours or filling out a template to complete an investigation does not result in high-quality analysis. I’ve observed three outcomes associated with incident analyses: pushing paper, technical teaching, and surfacing surprises.
Pushing Paper
Regulatory requirements or fiduciary responsibility may require your organization to produce reports. In these cases, the value of analysis is largely superficial: it’s part of a process to generate a paper trail. It’s a bureaucratic, defensive procedure. Incident analyses are written to be filed, not read.
Technical Teaching
Imagine you operate an ecommerce site whose API fell over as requests stacked up. Your analysis of this incident is an opportunity to update documentation and procedures that can address technical gaps in people’s understanding of the architecture. This is a way of building systems to be robust in the face of past known conditions.
The value of analysis seems ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access