97 Things Every SRE Should Know

Chapter 25. With Incident Response, Start Small

Thai Wood

Resilience Roundup

There’s a good chance that your incident response plan looks something like the following:

1. Someone gets paged (possibly you!)

2. ???

3. Fix it

That’s the plan that develops in many cases on its own. As your organization and systems grow—in the number of people that operate it, the number of people it serves, or its complexity—that plan no longer fits. As part of an SRE or Ops team, you can watch for some of these signs:

• You’re unsure how to start an incident.

• You don’t know how or when to get more people involved.

• You don’t know whether to start a call or conference bridge or use chat.

• There’s no consistent way to notify people who might be affected by the incident.

• When battling the incident, it’s unclear who is doing what.

An incident by its very nature is a surprising event, a cognitively difficult task. Not being able to answer these questions introduces further uncertainty to an incident and can be incredibly costly. Instead of trying to investigate and solve the mystery of the incident directly, responders are trying to answer those questions by figuring out how to coordinate from scratch instead of attacking the problem. The results are split focus by the responders and longer downtimes.

Taking steps to change this pattern can be difficult. It can feel impossible to find the time or space to learn ...