Chaos Engineering

Chapter 3. Hypothesize about Steady State

For any complex system, there are going to be many moving parts, many signals, and many forms of output. We need to distinguish in a very general way between systemic behaviors that are acceptable and behaviors that are undesirable. We can refer to the normal operation of the system as its steady state.

If you are developing or operating a software service, how do you know if it is working? How do you recognize its steady state? Where would you look to answer that question?

Steady State

The Systems Thinking community uses the term “steady state” to refer to a property such as internal body temperature where the sys‐ tem tends to maintain that property within a certain range or pat‐ tern. Our goal in identifying steady state is to develop a model that characterizes the steady state of the system based on expected values of the business metrics. Keep in mind that a steady state is only as good as its customer reception. Factor service level agreements (SLAs) between customers and services into your definition of steady state.

If your service is young, perhaps the only way you know that everything is working is if you try to use it yourself. If your service is accessible through a website, you might check by browsing to the site and trying to perform a task or transaction.

This approach to checking system health quickly reveals itself to be suboptimal: it’s labor-intensive, which means we’re less likely to do it. We can automate these kinds ...