Designing Data-Intensive Applications, 2nd Edition

Chapter 9. The Trouble with Distributed Systems

They’re funny things, Accidents. You never have them till you’re having them.

A.A. Milne, The House at Pooh Corner (1928)

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 9th chapter of the final book. The GitHub repo for this book is https://github.com/ept/ddia2-feedback.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out on GitHub.

As discussed in “Reliability and Fault Tolerance”, making a system reliable means ensuring that the system as a whole continues working, even when things go wrong (i.e., when there is a fault). However, anticipating all the possible faults and handling them is not that easy. As a developer, it is very tempting to focus mostly on the happy path (after all, most of the time things work fine!) and to neglect faults, since they introduce a lot of edge cases.

If you want your system to be reliable in the presence of faults you have to radically change your mindset, and focus on the things that could go wrong, even though they may be unlikely. It doesn’t matter whether there is only a one-in-a-million chance of a thing going wrong: in a large enough system, one-in-a-million events happen every day. Experienced systems operators will tell you that ...