Frank Kane's Taming Big Data with Apache Spark and Python

Parsing (mapping) the input data

Our first step is just to parse our input data into what we need; nothing special here, we're going to start by creating a lines RDD as shown here. This is calling textFile on our SparkContext object with our source data, and that's just going to give us an RDD where every individual line of that comma-separated value list is an individual entry in our RDD:

lines = sc.textFile("file:///SparkCourse/fakefriends.csv") 

Now things get interesting; I'm going to transform my lines RDD into an rdd RDD (very creatively named) by calling map on it. Also, I'm passing in the parseLine function to actually conduct that mapping:

rdd = lines.map(parseLine) 

So every line from my lines RDD will be passed into parseLine ...