Building Secure and Reliable Systems

Chapter 18. Recovery and Aftermath

By Alex Perry, Gary O’Connor‎, and Heather Adkins

with Nick Soda

To avoid service disruptions for your users, you need to be able to quickly recover from security- and reliability-related incidents. However, there’s a key difference when you are recovering from a security incident: your attacker. A persistent attacker can leverage ongoing access to your environment or reengage at any moment, even while you’re executing a recovery.

In this chapter, we take a deep dive into what people designing, implementing, and maintaining systems need to know about recovering from attacks. The people performing recovery efforts often aren’t security professionals—they’re the people who build the affected systems and operate them every day. The lessons and examples in this chapter highlight how to keep your attacker at bay while you’re recovering. We walk through the logistics, timeline, planning, and initiation of the recovery phases. We also discuss key tradeoffs, like when to disrupt an attacker’s activity versus allowing them to remain on your systems so you can learn more about them.

If your organization experiences a serious incident, will you know how to recover? Who performs that recovery, and do they know what decisions to make? Chapter 17 of the SRE book and Chapter 9 of the SRE workbook discuss practices for preventing and managing service outages. Many of those practices are also relevant to security, but recovering from security attacks has unique ...