Learning Chaos Engineering

Chapter 11. Human Intervention in Chaos Experiment Automation

So far you’ve created automated chaos engineering experiments that run from beginning to end. You kick off the experiment with chaos run, and when it is completed, you look at the output to see if there has been any deviation from the steady-state hypothesis that might indicate evidence of a possible system weakness.

For many automated chaos experiments, this execution strategy is perfect. Especially when you consider running chaos experiments continuously as tests (see Chapter 12), the capability of running experiments with no human intervention is essential.

Sometimes, though, more operational control is needed over the execution of a chaos experiment. Perhaps you have a probe activity in the steady-state hypothesis that you’d like to occasionally skip, or maybe there’s an action activity in the experiment’s method that you’d like to get a choice between continuing and executing or skipping in some cases. You might also want to introduce the ability to hit a metaphorical “Big Red Stop Button” so that you can choose to abort the experiment entirely if things start to go awry.

Each of these cases requires human intervention in the execution of your chaos experiments. The experiments will still be automated, but you want to allow someone to make choices about how an experiment is executed as it is running. All of these scenarios can be accomplished using the Chaos Toolkit’s control feature.

Let’s see how to do this now ...