Frank Kane's Taming Big Data with Apache Spark and Python

Examining the min-temperatures script

We start off with the usual boilerplate stuff, importing what we need from pyspark and setting up a SparkContext object that we're going to call MinTemperatures:

from pyspark import SparkConf, SparkContext 
 
conf = SparkConf().setMaster("local").setAppName("MinTemperatures") 
sc = SparkContext(conf = conf) 

If you skip down to line 13, you can see that we're loading up our source data file from 1800.csv into a lines RDD:

lines = sc.textFile("file:///SparkCourse/1800.csv") 

In line 14, we then parse that out using our parseLine mapper function:

parsedLines = lines.map(parseLine) 

We defined that function here:

 def parseLine(line): fields = line.split(',') stationID = fields[0] entryType = fields[2] temperature ...