Although an outage is a terrifying prospect, you should embrace it as an opportunity. Failure can expand and improve your understanding of your systems.
Three years ago, Indeed suffered one of the worst outages in its history. No single fault or failure caused this outage. Rather, it was a complex interaction of bugs, design decisions, capacity loss, and poor situational awareness during incident response. Indeed learned valuable lessons from this event. It identified ways to make the systems more resilient and improved the approach to the incident lifecycle within the engineering culture.
Alex Elman uses the narrative of this incident to demonstrate how a site-wide outage can inform increased resilience and reduced operational complexity. Learning from failure is a feedback loop rather than a one-off process. He applies Indeed’s outage as a practical example of what an iteration of this loop can look like. He shares with other SREs the success that has risen from this failure. Indeed hasn’t had a global site outage in the three years since this event.
Alex begins with a discussion of failure to set the stage for delivering the incident background, then discusses incident response and situational awareness. He explains conducting incident postmortems and learning from failure and designing for reliability, including resilience patterns such as circuit breaking and graceful degradation. Finally, he gets into resilience testing, running chaos tests, and closing the feedback loop, leaving some time for a question and answer session.
This session was recorded at the 2019 O'Reilly Velocity Conference in San Jose.
- Title: Learning from failure: Why a total site outage can be a good thing
- Release date: October 2019
- Publisher(s): O'Reilly Media, Inc.
- ISBN: 0636920338437
You might also like
Software Architecture Fundamentals, Second Edition
Being a successful software architect is more than just possessing technical knowledge. It’s about thinking like …
Software Engineering at Google
Today, software engineers need to know not only how to program effectively but also how to …
Head First Design Patterns, 2nd Edition
You know you don’t want to reinvent the wheel, so you look to design patterns—the lessons …
Microservices Patterns Video Edition
"A comprehensive overview of the challenges teams face when moving to microservices, with industry-tested solutions to …