O'Reilly logo

Chaos Engineering Observability by Russ Miles

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 2. Chaos Experiment Signals

“Data! Data! Data!” he cried impatiently. “I can’t make bricks without clay.”

Sherlock Holmes, from “The Adventure of the Copper Beeches” by Sir Arthur Conan Doyle

Observability feeds on the signals that a system emits that provide the raw data about the system’s behavior. Observability is limited by the signals that a system puts out.

As a chaos experiment executes, it can emit a number of different signals that are useful to system observability. In this chapter you’re going to see how the Chaos Toolkit supports two possible mechanisms for producing observability signals from automated chaos experiment execution:

Chaos Notifications

Coarse-grained, flow-level notifications during an experiment’s execution.

Chaos Controls

Fine-grained listeners, and even influencers, of an experiment’s execution.

Coarse-Grained Signals Through Notifications

Chaos experiment notifications are emitted by the Chaos Toolkit when an experiment is executed, as shown in Figure 2-1.

An image of the notifications available from an experiment's execution.
Figure 2-1. The notifications available from a chaos experiment’s execution

Chaos experiment notifications are coarse-grained because they are only triggered at the highest level of an experiment’s execution. The Slack extension for the Chaos Toolkit uses chaos notifications to then surface those signals in a specific Slack channel.

Once the Slack extension is installed, add this ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required