Frank Kane's Taming Big Data with Apache Spark and Python

Step 1 - Implement countByValue() the hard way to create a new RDD

So the first thing we're going to do is actually implement what countByValue does by hand, the hard way. This way we can actually play with the results more directly and stick the results in an RDD instead of just getting a Python object that we need to deal with at that point. The way we do that is we take our map of words-words.map-and we use a mapper that just converts each individual word into that word and the value of 1-(lambda x: (x, 1)):

wordCounts = words.map(lambda x: (x, 1)).reduceByKey(lambda x, y: x + y) 

This is a very similar trick to what we did in the earlier script where we computed the average number of friends by age in a fake social network. There we ...