“Come, Watson, come!"” he cried. “The game is afoot. Not a word! Into your clothes and come!”
Sherlock Holmes, from “The Return of Sherlock Holmes” by Sir Arthur Conan Doyle
In Chapter 2 you were introduced to the sorts of chaos experiment observability signals that a chaos experiment’s execution may provide. Now it’s time to look at how those signals can be turned into something useful to your own observability picture.
Centralized logging systems are widely recognized as a foundational part of any system’s observability toolkit. By bringing all of the log events of a system together in one place, you are able to interrogate, inspect, correlate, and begin to comprehend what happened and when across your system. Now you’re going to see how you can convert raw signals from your running chaos experiments to send them as valuable log events to a centralized logging system.
The Chaos Toolkit open source community has created an implementation of a Control (see “Fine-Grained Signals Through Chaos Controls”) that bridges from a running chaos experiment to a centralized logging system.
The following code sample, taken from a full Logging Control shows how you can implement a Chaos Toolkit Control function to hook into the lifecycle of a running chaos experiment:
# Send the experiment