The previous chapter described a number of patterns for splitting
and chaining queues together to achieve more complex batch processing.
Duplicating and producing multiple different outputs is often
an important part of batch processing, but sometimes it is equally
important to pull multiple outputs back together in order to
generate some sort of aggregate output. A generic illustration
of such a pattern is shown in Figure 12-1.
Figure 12-1. A generic parallel work distribution and result aggregation batch system
Probably the most canonical example of this aggregation is the reduce
part of the MapReduce pattern. It’s easy to see that the map step
is an example of sharding a work queue, and the reduce step is an
example of coordinated processing that eventually reduces a large number
of outputs down to a single aggregate response. However, there are
a number of different aggregate patterns for batch processing, and this
chapter discusses a number of them in addition to real-world applications.
Join (or Barrier Synchronization)
In previous chapters, we saw patterns for breaking up work and distributing it in parallel on multiple nodes. In particular, we saw how a sharded work queue could distribute work in parallel to a number of different work queue shards. However, sometimes when processing a workflow, it is necessary to have the complete ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month, and much more.
