Frank Kane's Taming Big Data with Apache Spark and Python

Examining the script

So let's just review again what's going on here. We start off with the usual boilerplate stuff. We import what we need from pyspark for Spark:

from pyspark import SparkConf, SparkContext 

We then set up a SparkContext object that's just going to run locally on our own computer on one process and we're going to call it FriendsByAge:

conf = SparkConf().setMaster("local").setAppName("FriendsByAge") 
sc = SparkContext(conf = conf) 

Alright, so the first thing we do is pretty typical. We call textFile on the SparkContext object to parse our input data, our fakefriends.csv file, and if you do have that in some other folder, make sure you modify that here first:

lines = sc.textFile("file:///SparkCourse/fakefriends.csv")