97 Things Every SRE Should Know

Chapter 39. The Maestros of Incident Response

Andrew Louis

DigitalOcean

We’ve all been there: the first time we’re the IMOC (Incident Manager On-Call, or Incident Commander, others might call it). My first IMOC page hit a year into the gig and, regardless of all the observing I did before, my handling of it paled compared to the performances before mine. It wasn’t my last fumble, but I began to build a high-level framework for incident management. With each fumble that followed, I added something new to it. The framework has remained valuable as a starting point, and I hope it will be helpful to you too.

There is more and better material dedicated to expanding how to manage an incident, but here are the primary principles that I keep at the forefront.

Stop the Bleeding

Keep the focus unrelentingly on prioritizing mitigation. Although the conversation might drift into deep root-cause investigations and discussions of longer-term solutions, the first impulse should be to keep the ongoing conversation focused solely on recovering the current situation.

What’s Everyone Doing?

At regular intervals (be wary of the cost this could impose on folks working on the problem), continue to raise the question of what everyone is doing. The goal here will be to keep track of the efforts, prevent overlapping work from going on, and get health checks from the parties involved.

Raising this question ...