Chaos Engineering Observability

Chapter 4. Tracing Chaos Experiments

“Nothing clears up a case so much as stating it to another person.”

Sherlock Holmes, from “Silver Blaze” by Sir Arthur Conan Doyle

Distributed tracing is critical to comprehending how an interaction with a running system propagates across the system. By enriching your logging messages with trace information, you can piece together the crucial answers to questions such as what happened, in what order, and who instigated the whole thing. When it comes to understanding how chaos experiments affect a whole system, add your chaos experiments to the tracing observability picture.

In this chapter you’re going to see how you can use the raw observability signals from the Chaos Toolkit (Chapter 2) to enable a new type of Control that will be able to push trace information into distributed tracing dashboards so that you can view your chaos experiment traces alongside your regular system interaction traces.

Open Tracing

Open Tracing is a helpful open standard for adding and communicating distributed tracing about a system.

The Chaos Toolkit comes with an Open Tracing extension that provides an Open Tracing Control, and it’s this control that you are going to use and see in action in this chapter.

The Open Tracing Control

After you have installed the Open Tracing Chaos Toolkit extension, your experiments can be configured to use the open tracing control by specifying a configuration block:

{
    "configuration": {
        "tracing_provider": "jaeger",
        "tracing_host" ...