Security Chaos Engineering

Chapter 5. Operating and Observing

There is nothing staid, nothing settled, in this universe. All is rippling, all is dancing; all is quickness and triumph.

Virginia Woolf, The Waves

The operations phase of software delivery deals with managing and studying the system while it runs in production. Production is where the system interacts with real customers and users. Much like rehearsals of a play, all the other phases build to this one. Once the software we design, build, and deploy is delivered to end users and is prancing in production environments, it can finally deliver value for the organization. For any organization with digitally delivered products and services, this phase is where the money printer is turned on (and goes brrrr!).

The operating and observing phase is where our mental models encounter their challenger: reality. It’s tempting to only observe what you expect to encounter; think back to the Jurassic Park example from Chapter 1 where they only checked for the number of dinosaurs they expected, not anticipating that there could be more than in their mental models. As we shepherd our systems in the tumultuous pastures of the internet, we must continually refine our mental models—keeping an open mind that our assumptions might be wrong and being curious enough to seek out evidence. That is, we need the last two ingredients in the recipe: feedback loops and ...