Chaos Engineering Observability

Chapter 3. Logging Chaos Experiments

“Come, Watson, come!"” he cried. “The game is afoot. Not a word! Into your clothes and come!”

Sherlock Holmes, from “The Return of Sherlock Holmes” by Sir Arthur Conan Doyle

In Chapter 2 you were introduced to the sorts of chaos experiment observability signals that a chaos experiment’s execution may provide. Now it’s time to look at how those signals can be turned into something useful to your own observability picture.

Centralized logging systems are widely recognized as a foundational part of any system’s observability toolkit. By bringing all of the log events of a system together in one place, you are able to interrogate, inspect, correlate, and begin to comprehend what happened and when across your system. Now you’re going to see how you can convert raw signals from your running chaos experiments to send them as valuable log events to a centralized logging system.

From Signals to Centralized Logging

The Chaos Toolkit open source community has created an implementation of a Control (see “Fine-Grained Signals Through Chaos Controls”) that bridges from a running chaos experiment to a centralized logging system.

The following code sample, taken from a full Logging Control shows how you can implement a Chaos Toolkit Control function to hook into the lifecycle of a running chaos experiment:

...

def before_experiment_control(context: Experiment, secrets:
                              Secrets):
# Send the experiment

    if not with_logging.enabled:
        return

    event = {
        "name" ...