Site Reliability Engineering, 2nd Edition

Chapter 16. Designing for Resilience

By Vitaliy Shipitsyn

with Christof Leng and Garrett Holthaus

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 17th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

Resilience enhances reliability. Reliability is a system’s ability to consistently operate within designed norms, for functional and non-functional requirements. Resilience, on the other hand, is when systems can detect and diagnose active failure patterns automatically or with the help of trained people or AI agents. A resilient system uses pre-defined response plans to reduce the impact of those failures, even for novel failures that were not anticipated at system design.

Resilience provides business continuity for the users who rely on your systems. A reliable system may consistently meet its SLO targets under normal conditions, but a resilient system maintains high levels of reliability in more adverse conditions. When a pre-defined response isn’t prepared, that alone might burn through your system’s error budgets. A lasting outage without a rebounding clean recovery can have an existential ...