Day 1: MapReduce
MapReduce is a broad term. Sometimes it’s used to describe the common pattern of breaking an algorithm down into two steps: a map over a data structure, followed by a reduce operation. Our functional word count (see the code) is an example of exactly this (remember that frequencies is implemented using reduce). As we saw in Day 2: Functional Parallelism, one of the benefits of breaking an algorithm down in this way is that it lends itself to parallelization.
But MapReduce can also be used to mean something more specific—a system that takes an algorithm encoded as a map followed by a reduce and efficiently distributes it across a cluster of computers. Not only does such a system automatically partition both the data and ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access