Frank Kane's Taming Big Data with Apache Spark and Python

Calling reduceByKey

If it's not greater than zero, we'll keep on processing. We will call the reduceByKey function to actually gather together all of the nodes that we might have generated in that flatMap operation and recombine them together for each given character ID. There can be only one node per character ID, and this enforces that:

    iterationRdd = mapped.reduceByKey(bfsReduce) 

The bfsReduce function, as you can see up at line 64, simply gathers the nodes back together:

 def bfsReduce(data1, data2): edges1 = data1[0] edges2 = data2[0] distance1 = data1[1] distance2 = data2[1] color1 = data1[2] color2 = data2[2] distance = 9999 color = 'WHITE' edges = [] # See if one is the original node with its connections. # If so preserve them. ...