ZeroMQ

Robust Reliable Queuing (Paranoid Pirate Pattern)

The Simple Pirate queue pattern works pretty well, especially because it’s just a combination of two existing patterns. Still, it does have some weaknesses:

• It’s not robust in the face of a queue crash and restart. The client will recover, but the workers won’t. While ØMQ will reconnect workers’ sockets automatically, as far as the newly started queue is concerned the workers haven’t signaled ready, so they don’t exist. To fix this we have to do heartbeating from queue to worker so that the worker can detect when the queue has gone away.

• The queue does not detect worker failure, so if a worker dies while idle, the queue can’t remove it from its worker queue until the queue sends it a request. The client waits and retries for nothing. It’s not a critical problem, but it’s not nice. To make this work properly, we need to do heartbeating from worker to queue, so that the queue can detect a lost worker at any stage.

We’ll fix these issues in a properly pedantic Paranoid Pirate pattern.

We previously used a REQ socket for the worker. For the Paranoid Pirate worker, we’ll switch to a DEALER socket (Figure 4-3). This has the advantage of letting us send and receive messages at any time, rather than the lock-step send/receive that REQ imposes. The downside of DEALER is that we have to do our own envelope management (re-read Chapter 3 for background on this concept).

Figure 4-3. The Paranoid Pirate pattern

We’re still using the Lazy Pirate client. ...