97 Things Every SRE Should Know

Chapter 41. If You’re Doing Runbooks, Do Them Well

Spike Lindsey

Shopify

Runbooks (also known as playbooks) are not a silver bullet (nothing is), and yet a common action item after an incident is “update runbooks/add missing runbooks.” So why do we persist?

Runbooks are generally concerned with known unknowns, and we cannot anticipate every problem. They share all of documentation’s pitfalls: accuracy, quality, maintainability, drift. They are relied on in time-pressured, stressful situations. They can sometimes be a fragile bandage over reliability issues: instead of investing in fixing underlying issues, teams overinvest in runbooks, creating new sources of toil. They can be seen as direct substitutes for training and experience, resulting in situations in which newer or less-experienced team members are put on call for systems that they don’t yet understand, simply because “runbooks exist.” Inaccurate or outdated runbooks can sometimes be more dangerous than no runbooks.

From that brief list, you might think that I am deeply opposed to runbooks—quite the opposite. As with any system, understanding the downsides means that we can work to mitigate or fix them and benefit better from the upsides.

Runbook creation, maintenance, and review should be a whole-team activity. Recognizing that runbooks are a solution to a knowledge distribution problem (rather than a technical problem) ...