Frank Kane's Taming Big Data with Apache Spark and Python

Using a mapper and a reducer

Now, to do that, we're basically going to create a map function that will go through the data looking for gray nodes. If a node is gray, our function will create new nodes that have a distance incremented by one and a color of gray. These new nodes will have no connections, we'll combine them back together with their connection list in the reducer. Obviously, this is going to be a flatMap operation because we might be producing multiple nodes for each input node. We then color the processed node black, and we will copy the node itself into the results returned by the flatMap operation so we don't lose it.

We also use a reduce function. Our reducer recombines everything. It looks for nodes tied to the same hero ...