97 Things Every SRE Should Know

Chapter 49. Legendary

Elise Gale

Microsoft

Why are some outages forgotten as soon as they are mitigated, while others go on to become team legend?

I believe it’s because legendary stories follow the hero’s journey, a model used in literature to understand a diverse set of tales, from Moses to Harry Potter. Outages, like a hero’s journey, have three key parts: the call to action, the road of trials, and the return. I imagine on-call engineers embarking on an epic quest, and following these steps shows us what makes an incident story exciting—and, hopefully, gives us the insight we need to prevent it from happening again.

My legendary call to action begins with, “It was two nights before Christmas. . . .” The timing matters because, frankly, the issue would have been much less interesting any other day of the year, but I was new on the team and had to find the courage, on a company holiday, to page my boss for help.

The road of trials is often the meatiest part—this is when you diagnose and mitigate the situation. This phase is a gold mine for WTF moments. If you are telling your story aloud, this is where your audience should collectively groan. Maybe the log message was missing the one detail you needed to unravel the mystery. Or perhaps, like me, a simple configuration change took six hours and four developers to deploy because of missing tooling.

We discovered the issue was ...