Chaos Engineering

Chapter 11. People in the Loop

John Allspaw

The very notion that performing experiments on a software system can be a valuable approach to better understand its behavior is an implied acknowledgment its behavior cannot be comprehensively prescribed at the time the system is designed. Most modern software engineers understand this; writing software and understanding how it works (and how it might fail) are two very different things.

Developing and maintaining an understanding about a system’s behavior depends on people’s ability to revise or recalibrate their “mental models” of its behavior.¹ The process of experimentation in production systems can be seen (alongside post-incident analysis) as a fruitful opportunity for this recalibration or “mental model updating.”

Much of this book is dedicated to approaches, nuances, details, and perspectives on the how of Chaos Engineering. The very existence of this book and the topic’s presence in the current dialogue of software engineering and operations is exciting and is likely to continue evolving. It brings to the table the notion that confidence can be built through more sophisticated and mature ways.

As should be expected in the world of software, new ideas and debate have arisen about how to automate more and more in the context of Chaos Engineering. This tendency is understandable (given that building “automation” is essentially the raison d’être of software engineers). Proposals in this vein can be unaware of the irony of introducing ...