Chaos Engineering Observability

Chapter 2. Chaos Experiment Signals

“Data! Data! Data!” he cried impatiently. “I can’t make bricks without clay.”

Sherlock Holmes, from “The Adventure of the Copper Beeches” by Sir Arthur Conan Doyle

Observability feeds on the signals that a system emits that provide the raw data about the system’s behavior. Observability is limited by the signals that a system puts out.

As a chaos experiment executes, it can emit a number of different signals that are useful to system observability. In this chapter you’re going to see how the Chaos Toolkit supports two possible mechanisms for producing observability signals from automated chaos experiment execution:

Chaos Notifications

Coarse-grained, flow-level notifications during an experiment’s execution.

Chaos Controls

Fine-grained listeners, and even influencers, of an experiment’s execution.

Coarse-Grained Signals Through Notifications

Chaos experiment notifications are emitted by the Chaos Toolkit when an experiment is executed, as shown in Figure 2-1.

Figure 2-1. The notifications available from a chaos experiment’s execution

Chaos experiment notifications are coarse-grained because they are only triggered at the highest level of an experiment’s execution. The Slack extension for the Chaos Toolkit uses chaos notifications to then surface those signals in a specific Slack channel.

Once the Slack extension is installed, add this ...