A common problem you will hit when using the pub-sub pattern in real life is the slow subscriber. In an ideal world, we stream data at full speed from publishers to subscribers. In reality, subscriber applications are often written in interpreted languages, or do a lot of work, or are just badly written, to the extent that they can’t keep up with publishers.
How do we handle a slow subscriber? The ideal fix is to make the subscriber faster, but that might take a significant amount of work and time. Some of the classic strategies for handling a slow subscriber are:
Queue messages on the publisher. This is what Gmail does when I don’t read my email for a couple of hours. But in high-volume messaging, pushing queues upstream has the thrilling but unprofitable result of making publishers run out of memory and then crash—especially if there are lots of subscribers and it’s not possible to flush to disk for performance reasons.
Queue messages on the subscriber. This is much better, and it’s what ØMQ does by default if the network can keep up with things. If anyone’s going to run out of memory and crash, it’ll be the subscriber rather than the publisher, which is fair. This is perfect for “peaky” streams where a subscriber can’t keep up for a while, but can catch up when the stream slows down. However, it’s no answer to a subscriber that’s simply too slow in general.
Stop queuing new messages after a while. This is what Gmail does when my ...