Frank Kane's Taming Big Data with Apache Spark and Python

Setting up the SparkContext object

Next, we actually create our SparkContext, and this is going to look very similar in every Spark script that we write in Python. We have these two lines:

conf = SparkConf().setMaster("local").setAppName("RatingsHistogram") 
sc = SparkContext(conf = conf) 

So let's look at what's going on here. In the first line, we have the SparkConf object that we imported earlier and we're telling it to set its master node as the local machine. Basically, it says we're going to be running on the local box only, not on the cluster, just on this one system. There are extensions to local that tell you to actually split it up among multiple CPU cores that you might have on your machine, but this is a very simple example, so ...