Frank Kane's Taming Big Data with Apache Spark and Python

Adding up co-occurrence by hero ID

Now remember, heroes can span multiple lines, so next we need to call reduceByKey in order to actually add them all up together. In the case where we have more than one line for a given hero, our standard lambda function for combining things together in a reduce operation will just add them all up:

totalFriendsByCharacter = pairings.reduceByKey(lambda x, y : x + y) 

If we have two different entries for, say, superhero ID 5, it will get the counts for each, add them together, and reduce them into a single entry, here in line 20. So, we now we have totalFriendsByCharacter, which is itself a key/value RDD, where the key is the superhero ID and the values are the total number of co-occurrences seen for that ...