ZeroMQ

Scaling to Multiple Clusters

Now we scale this out to more than one cluster. Each cluster has a set of clients and workers, and a broker that joins these together, as illustrated in Figure 3-15.

Figure 3-15. Multiple clusters

The question is, how do we get the clients of each cluster talking to the workers of the other cluster? There are a few possibilities, each with pros and cons:

• Clients could connect directly to both brokers. The advantage here is that we don’t need to modify the brokers or workers. However, the clients get more complex and become aware of the overall topology. If we want to add a third or fourth cluster, for example, the clients are affected. In effect, we have to move routing and failover logic into the clients, and that’s not nice.

• Workers might connect directly to both brokers. But REQ workers can’t do that; they can only reply to one broker. We might use REPs, but REPs don’t give us customizable broker-to-worker routing like load balancing does, only the built-in load balancing. That’s a fail; if we want to distribute work to idle workers, we precisely need load balancing. One solution would be to use ROUTER sockets for the worker nodes. Let’s label this “Idea #1.”

• Brokers could connect to each other. This looks the neatest because it creates the fewest additional connections. We can’t add clusters on the fly, but that is probably out of our scope anyway. With ...