Building Resilient Distributed Systems

Chapter 3. Retries and Idempotency

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the fourth chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at [e-mail].

‘Tis a lesson you should heed: Try, try, try again. If at first you don’t succeed, Try, try, try again

W. E. Hickson

When a call fails, trying again makes a lot of sense. When you fail to establish a connection, it’s possible your request just happened to be routed to a machine that was overloaded, or a machine that just got shut down for maintenance. Perhaps you hit a gateway at a point it was congested, or your request was routed to a server that was slow to start up. In many of these cases, simply trying again may well see the subsequent request routed to a machine that can serve the request promptly.

If you do decide to retry, there are still a host of questions to answer. How many times should you retry? Should you wait between retries? And are there situations where carrying out a retry can cause issues?

Before I answer those questions though, let’s start at the beginning. Does it always ...