Learning Chaos Engineering

Chapter 12. Continuous Chaos

With every chaos experiment that you write and run, you increase your chances of finding evidence of dark debt that you can learn from and use to improve your system. Your chaos experiments will start out as explorations of your system; ways to ask yourself, “If this happens, I think the system will survive…or will it?” You’ll gradually build a catalog of experiments for your system that explores a selection of your Hypothesis Backlog, helping you build trust and confidence that you’re proactively exploring and surfacing weaknesses before they affect your users.

Some of your chaos experiments will then graduate into a different phase of their lives. The first phase of an experiment’s life, as just described, is about finding evidence of system weaknesses. It’s about exploring and uncovering that dark debt inherent in all complex sociotechnical systems. Over time, you will choose to overcome some or all of the weaknesses that your automated chaos experiments have surfaced evidence for. At that point, a chaos experiment enters the second phase of its life: it becomes a chaos test.

A chaos experiment is exploration; a chaos test is validation. Whereas a chaos experiment seeks to surface weaknesses and is celebrated when a deviation is found,¹ a chaos test validates that previously found weaknesses have been overcome.

There’s more good news: a chaos experiment and a chaos test look exactly the same. Only the interpretation of the results is different. ...