Chaos Engineering Observability

Preface

If you’re considering running chaos experiments to find system weaknesses, especially in production, then observability will be on your mind. This book is for everyone adopting automated chaos engineering in their teams and to ensure that they can execute those experiments as safely as possible by bringing those chaos experiments into the overall system observability picture.

This book introduces the concept of chaos observability: how to run chaos experiments and bring that work into your overall system observability picture. You will see how chaos engineering experiments leverage a system’s observability and contribute to it. This all begins by introducing the key concept of Chaos Experiment Observability Signals, covered in Chapter 2.

Throughout this book, high-level samples are shown using the free and open source Chaos Toolkit. Although only the Chaos Toolkit’s observability capabilities are shown, the hope is that this book will prompt the need for observability across other chaos engineering implementations, possibly resulting in a set of open standard concepts and guidelines for chaos observability.

This book provides code samples of integrations with OpenTracing, visualized using Jaeger, for tracing chaos experiments alongside distributed system traces, and centralized logging using Humio. These samples show concrete implementations of how any system could be integrated with available chaos experiment observability signals.

For more on chaos engineering, see Chaos Engineering by Ali Basiri, Nora Jones, Aaron Blohowiak, Lorin Hochstein, and Casey Rosenthal (O’Reilly). For an introduction to Observability, see Distributed Systems Observability by Cindy Sridharan (O’Reilly).

Conventions Used in This Book

The following typographical conventions are used in this book:

Italic

Indicates new terms, URLs, email addresses, filenames, and file extensions.

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.

Constant width bold

Shows commands or other text that should be typed literally by the user.

Note

This element signifies a general note.

With those caveats and limitations, and ideally a copy of the aforementioned books on hand, I hope you enjoy this book and I wish you:

“Happy (Observable) Chaos Engineering!”