Chaos Engineering Observability

Chapter 5. Conclusion

Observability and chaos engineering go hand in hand. As you explore chaos engineering, the observability of your system will have to improve as you ask important questions about how you make sense of your running system.

Any chaos introduced into a system also needs to participate in your system’s observability picture by contributing a set of chaos signals. The lifecycle of a chaos experiment offers a number of signals that can be channeled into your various observability systems.

In this book you’ve seen how logging systems, such as Humio, and Open Tracing can be integrated with from chaos experiments in the free and open source Chaos Toolkit to build the foundations of this chaos observability, but this is just the starting point!

The Notification and Control extension APIs in the Chaos Toolkit exist so that you can integrate your own chaos experiments into your own observability toolsets. More implementations are already planned for the Chaos Toolkit Incubator, and of course you can also create your own.

With observability added to your own chaos engineering experiments, they can contribute, like good system citizens, to your observability picture. Chaos experiments themselves should never be a surprise, although their findings can sometimes be. Making sure you have good system and chaos observability should make sure they never are.