Although an outage is a terrifying prospect, you should embrace it as an opportunity. Failure can expand and improve your understanding of your systems.
Three years ago, Indeed suffered one of the worst outages in its history. No single fault or failure caused this outage. Rather, it was a complex interaction of bugs, design decisions, capacity loss, and poor situational awareness during incident response. Indeed learned valuable lessons from this event. It identified ways to make the systems more resilient and improved the approach to the incident lifecycle within the engineering culture.
Alex Elman uses the narrative of this incident to demonstrate how a site-wide outage can inform increased resilience and reduced operational complexity. Learning from failure is a feedback loop rather than a one-off process. He applies Indeed’s outage as a practical example of what an iteration of this loop can look like. He shares with other SREs the success that has risen from this failure. Indeed hasn’t had a global site outage in the three years since this event.
Alex begins with a discussion of failure to set the stage for delivering the incident background, then discusses incident response and situational awareness. He explains conducting incident postmortems and learning from failure and designing for reliability, including resilience patterns such as circuit breaking and graceful degradation. Finally, he gets into resilience testing, running chaos tests, and closing the feedback loop, leaving some time for a question and answer session.
This session was recorded at the 2019 O'Reilly Velocity Conference in San Jose.
- Title: Learning from failure: Why a total site outage can be a good thing
- Release date: October 2019
- Publisher(s): O'Reilly Media, Inc.
- ISBN: 0636920338437
You might also like
Effective Incident Response Team, The
When an intruder, worm, virus, or automated attack succeeds in targeting a computer system, having specific …
How Slack rearchitected its system to achieve zero downtime and improve latency, reliability and availability
In 2016, Slack faced a problem: the load on its backend servers had increased by 1,000x. …
Site Reliability Engineering on AWS
Reliability in AWS includes the ability of a system to recover from infrastructure or service disruptions. …
The Site Reliability Workbook
In 2016, Googleâ??s Site Reliability Engineering book ignited an industry discussion on what it means to …