Node.js Design Patterns - Second Edition

Pipelines and task distribution patterns

In Chapter 9, Advanced Asynchronous Recipes, we learned how to delegate costly tasks to multiple local processes, but even though this was an effective approach, it cannot be scaled beyond the boundaries of a single machine. In this section, we are going to see how it's possible to use a similar pattern in a distributed architecture, using remote workers located anywhere in a network.

The idea is to have a messaging pattern that allows us to spread tasks across multiple machines. These tasks might be individual chunks of work or pieces of a bigger task split using a divide and conquer technique.

If we look at the logical architecture represented in the following figure, we should be able to recognize a familiar ...