Frank Kane's Taming Big Data with Apache Spark and Python

Parse (map) the input data

The first thing we need to do is load that data up into an RDD. So we'll call textFile on our SparkContext and load every line of that input into a lines RDD:

lines = sc.textFile("file:///SparkCourse/1800.csv") 

We'll then take that RDD that we called lines and apply our parseLine mapper function to it to create a parseLines RDD:

parsedLines = lines.map(parseLine) 

This is what we want that function to do:

def parseLine(line): 
    fields = line.split(',') 
    stationID = fields[0] 
    entryType = fields[2] 
    temperature = float(fields[3]) * 0.1 * (9.0 / 5.0) + 32.0 
    return (stationID, entryType, temperature) 

The function will split out our fields by commas:

fields = line.split(',') 

Then it'll extract the station ID text ...