97 Things Every SRE Should Know

Chapter 92. Risk and Rot in Sociotechnical Systems

Laura Nolan

 

We work in organizations made up of people, all subject to outcome bias and prone to underestimate or overestimate risks, depending on to what extent normalization of deviance has set in on our team. Executives can become far removed from the reality of life at the front line, and their appreciation of probabilities of adverse events can be strongly affected by recent outcomes.

There is a phenomenon in operations that I’ve heard called the paradox of preparation—an organization that is effectively managing risks and preventing problems can fail to be recognized as such. Bad outcomes aren’t actually occurring because of this preventive work, so decision makers may come to believe that the risks are significantly lower than they actually are. Therefore, leaders may conclude that the organization that is preventing the negative events from occurring isn’t an efficient use of resources anymore.

One of the major functions of an SRE or operations team is to manage risks proactively. This kind of work covers a broad spectrum, from keeping systems patched, rotating certs and tokens, and validating backups, to less-routine things like writing runbooks and recovery tools, running disaster tests, performing production readiness reviews for new systems, and doing thorough reviews of near-miss production incidents. These are ...