Chapter 14. Incident Management and Post-Incident Response
You may encounter many defeats, but you must not be defeated. In fact, it may be necessary to encounter the defeats, so you can know who you are, what you can rise from, how you can still come out of it.
Maya Angelou
Incident response has two clear goals. The first is to get the system up and running as quickly as possible. The second is to capture enough information to identify ways in which the system (in a sociotechnical sense) can learn and adapt.
Broadly speaking, this chapter is split into two parts. First, I’ll look at the process of handling an incident and its immediate aftermath. Second, I’ll look at the process of post-incident review, and at how to make sure you create the best opportunities to learn from what happens.
I also want to be clear that this chapter is not intended to be an exhaustive discussion of incident management. Where applicable, I’ve added references to other resources from which you can learn more. My main goal in this chapter is to help you see incident management as an extension of the ideas of resilience engineering as a whole, a process in which dealing with surprise and continual learning is key to building more resilient systems. I’ll also show how incident management doesn’t follow a simple, structured workflow—like our distributed systems, it can be a messy, multifaceted affair.
Before that, though, we should probably define what an incident is. Oddly enough, this is not as clear-cut ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access