97 Things Every SRE Should Know

Chapter 97. Incidents: A Window into Gaps

Lorin Hochstein

Netflix

Incidents force us to confront the reality that our systems don’t always behave as we expect. It’s an uncomfortable feeling, and we want to have faith that this won’t happen again. To renew that faith after an incident, we work to determine why the system broke, but we can learn more from incidents than just how to prevent a repeat. We can identify a range of gaps—deficiencies that we can potentially address—that exist inside our organizations.

One kind of gap we can recognize is a tooling gap, when we see engineers having trouble using operational tools. For example, we can identify when engineers struggle to make sense of feedback from a tool during an incident, or when they make an error using a tool to make a production change. In particular, whenever we notice a workaround, someone performing a task the wrong way, that’s a clue there’s a tooling gap. Information about tooling gaps should be fed back to the owners of these operational tools.

Another kind of gap is an operational expertise gap. This is when engineers are missing important operational skills they need to do their jobs effectively. Perhaps a certain graph on a dashboard was always wrong, because the engineer who created it didn’t fully understand the metrics query language. It was only after the incident that somebody noticed. An operational expertise ...