Building Resilient Distributed Systems

Chapter 4. Retries and Idempotency

’Tis a lesson you should heed: Try, try, try again. If at first you don’t succeed, Try, try, try again.

William E. Hickson

When a call fails, trying again makes a lot of sense. When you fail to establish a connection, it’s possible your request just happened to be routed to a machine that was overloaded or to a machine that just got shut down for maintenance. Perhaps you hit a gateway at a point where it was congested, or maybe your request was routed to a server that was slow to start up. In many of these cases, simply trying again may well see the subsequent request sent to a machine that can serve the request promptly.

If you do decide to retry, there are still many questions to answer. How many times should you retry? Should you wait between retries? And are there situations in which carrying out a retry can cause problems? We’ll explore all these issues in this chapter.

Before I answer those more detailed questions, though, let’s start at the beginning and introduce the retry pattern properly.

Pattern: Retry Remote Operation

Reattempt an operation on a remote endpoint when you cannot confirm that the previous attempt succeeded.

If you call a service and ask it to do something and then are told that the operation failed, you may want to retry the operation. The word may here is important. Sometimes when an operation fails, the nature of the failure means that a retry doesn’t make sense. For example, if the operation was to see the state ...