Although an outage is a terrifying prospect, you should embrace it as an opportunity. Failure can expand and improve your understanding of your systems.
Three years ago, Indeed suffered one of the worst outages in its history. No single fault or failure caused this outage. Rather, it was a complex interaction of bugs, design decisions, capacity loss, and poor situational awareness during incident response. Indeed learned valuable lessons from this event. It identified ways to make the systems more resilient and improved the approach to the incident lifecycle within the engineering culture.
Alex Elman uses the narrative of this incident to demonstrate how a site-wide outage can inform increased resilience and reduced operational complexity. Learning from failure is a feedback loop rather than a one-off process. He applies Indeed’s outage as a practical example of what an iteration of this loop can look like. He shares with other SREs the success that has risen from this failure. Indeed hasn’t had a global site outage in the three years since this event.
Alex begins with a discussion of failure to set the stage for delivering the incident background, then discusses incident response and situational awareness. He explains conducting incident postmortems and learning from failure and designing for reliability, including resilience patterns such as circuit breaking and graceful degradation. Finally, he gets into resilience testing, running chaos tests, and closing the feedback loop, leaving some time for a question and answer session.
This session was recorded at the 2019 O'Reilly Velocity Conference in San Jose.
- Title: Learning from failure: Why a total site outage can be a good thing
- Release date: October 2019
- Publisher(s): O'Reilly Media, Inc.
- ISBN: 0636920338437
You might also like
40 Algorithms Every Programmer Should Know
Learn algorithms for solving classic computer science problems with this concise guide covering everything from fundamental …
O'Reilly Software Architecture Conference 2020 - New York, NY
O’Reilly Media’s Software Architecture Conference (SACON) New York 2020 was Maggie Carrol’s (MAG Aerospace) compelling examination …
Kubernetes for Absolute Beginners - Hands-on
Learn Kubernetes in simple, easy and fun way with hands-on coding exercises. For beginners in DevOps. …
O'Reilly Software Architecture Conference 2019 - New York, New York
What do you really need to consider when adopting a microservices architecture? How do you unite …