Chapter 11. Progressive Collapse
As I’ve touched on a few times in this book, as you increase the complexity of a system to improve one aspect of its resiliency, that new complexity can introduce new sources of failure all on its own. This is partly because as you introduce new components into a system, they bring their own challenges. But also, as a system becomes more complex and interconnected, it can become difficult to assess the impact of a seemingly small thing going wrong.
How can an overly tightened nut cause a tower block to collapse? How could a simple race condition cause thousands of systems to fail? In this chapter, we’ll look at the phenomenon of progressive collapse and examine what you can do to protect yourself from it.
What Is Progressive Collapse?
When the failure of a single component results in the failure of other connected elements, this is known as a progressive collapse. In the words of an NIST report on reducing the potential for progressive collapse in buildings:
The underlying characteristic of progressive collapse is that the final state of failure is disproportionately greater than the failure that initiated the collapse.1
In other words, the end result of the progressive collapse is much worse than would have been expected based on what initially went wrong.
In this chapter, we will mostly look at this concept in the context of our distributed systems. Before we do that, however, it’s useful to look at how this plays out in the context of physical ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access