Frank Kane's Taming Big Data with Apache Spark and Python

What Spark can do with key/value data?

As I mentioned earlier, one of the most useful things you can do with the key/value RDD is reduceByKey. What that does is to combine all the values that are found for the same key/value, using some function that you define. So, for example, if I want to add up all of the values for a given key, let's say all of the numbers of friends for a given age, something like this would do the job:

rdd.reduceByKey(lambda x, y: x + y) 

When you call a reduce function in Spark, it will get not just an x value but also an x and a y value. You can actually call those values whatever you want, but what you need to do is define a function that defines how to combine things together that are found for the same key. In ...