Site Reliability Engineering, 2nd Edition

Chapter 17. Designing for Resilience

By Vitaliy Shipitsyn with Christof Leng, Garrett Holthaus, and Kieran Broadfoot

Resilience enhances reliability. Reliability is a system’s ability to consistently operate within designed norms, meeting both functional and nonfunctional requirements. Resilience, on the other hand, is a system’s ability to detect and diagnose active failure patterns automatically or with the help of trained people or AI agents. A resilient system uses predefined response plans to reduce the impact of failures, even for novel failures that were not anticipated during system design.

Resilience provides business continuity for the users who rely on your systems. A reliable system may consistently meet its SLO targets under normal conditions, but a resilient system maintains high levels of reliability in more adverse conditions. When a predefined response isn’t prepared, that alone might burn through your system’s error budgets. A lasting outage without a rebounding clean recovery can have an existential impact on business survivability. If you lose all of your users’ data and cannot restore it, this may put your entire business at risk.

Your challenge is to anticipate unusual circumstances that stress systems beyond their original design expectations, testing their robustness. A system’s availability, security, integrity, or performance characteristics might fail—independently or simultaneously, partially or fully, for benign or malicious reasons. The open-ended ...