Learning Data Mining with Python - Second Edition

The intuition behind MapReduce

MapReduce has two main steps: the Map step and the Reduce step. These are built on the functional programming concepts of mapping a function to a list and reducing the result. To explain the concept, we will develop code that will iterate over a list of lists and produce the sum of all numbers in those lists.

There are also shuffle and combine steps in the MapReduce paradigm, which we will see later.

To start with, the Map step takes a function and applies it to each element in a list. The returned result is a list of the same size, with the results of the function applied to each element.

To open a new Jupyter Notebook, start by creating a list of lists with numbers in each sublist:

 a = [[1,2,1], [3,2], [4,9,1,0,2]] ...