97 Things Every SRE Should Know

Chapter 96. A Word from Software Safety Nerds

J. Paul Reed

Netflix

Any SRE tasked with the care and feeding of even the most modest of services will, inevitably, have a Bad Day™. After the incident plays out, we find ourselves party to a postmortem. Operational retrospectives—a more accurate name for what we software developers and operations engineers practice, unless your outage resulted in actual death—are likely not new to you. What may be news is the interest in the concept of safety and the mechanics of how we learn from incidents in software as individuals, teams, and whole organizations.

Following are a few insights we software safety nerds have uncovered and are actively studying, attempting to help us all learn more from these impactful events:

Leaning into complexity

The rise of web-scale distributed systems over the past 15 years has renewed interest in understanding the true impact of complexity science on our field. Many SREs throw the word around colloquially, but it was Mark Burgess’s configuration management work, described in In Search of Certainty, that laid the groundwork for us to consider that our systems, especially in the cloud, have more in common with quantum physics than with physics grounded in equal-and-opposite, cause-and-effect reactions.

The implications of quantum

Once we accept the true nature of the complexity of our systems, this calls into ...