Slow Subscriber Detection (Suicidal Snail Pattern)
A common problem you will hit when using the pub-sub pattern in real life is the slow subscriber. In an ideal world, we stream data at full speed from publishers to subscribers. In reality, subscriber applications are often written in interpreted languages, or do a lot of work, or are just badly written, to the extent that they canât keep up with publishers.
How do we handle a slow subscriber? The ideal fix is to make the subscriber faster, but that might take a significant amount of work and time. Some of the classic strategies for handling a slow subscriber are:
Queue messages on the publisher. This is what Gmail does when I donât read my email for a couple of hours. But in high-volume messaging, pushing queues upstream has the thrilling but unprofitable result of making publishers run out of memory and then crashâespecially if there are lots of subscribers and itâs not possible to flush to disk for performance reasons.
Queue messages on the subscriber. This is much better, and itâs what ÃMQ does by default if the network can keep up with things. If anyoneâs going to run out of memory and crash, itâll be the subscriber rather than the publisher, which is fair. This is perfect for âpeakyâ streams where a subscriber canât keep up for a while, but can catch up when the stream slows down. However, itâs no answer to a subscriber thatâs simply too slow in general.
Stop queuing new messages after a while. This is ...
Get ZeroMQ now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
