Frank Kane's Taming Big Data with Apache Spark and Python

Map example

Here's an example of a map operation, just to make these explanations a little more real. Let's say we have an RDD that's just, you know, hard-coded to contain the values 1, 2, 3, and 4:

rdd=sc.parallelize([1,2,3,4]) 

In Python, I could just say rdd.map and pass this lambda function of the x points to x times x:

rdd.map(lambda x:x*x) 

This one line will go ahead and square every value of that original RDD and then will return a new RDD that contains the values 1, 4, 9, and 16, which are the squares of 1, 2, 3, and 4.

What was that lambda thing? Well, if you're new to Python, this might be a new concept for you. Basically, a lot of Spark is centered around this concept called functional programming; it sounds fancy, but it's ...