Frank Kane's Taming Big Data with Apache Spark and Python

Examining the script

You can see we're importing the usual stuff at the top of the script. We do need to import the square root function from the math package in Python as well, so that we can do our similarity metric and compute similarities later on:

import sys 
from pyspark import SparkConf, SparkContext 
from math import sqrt 

Let's get down to the meat of it; down in line 42 we create our SparkContext:

conf = SparkConf().setMaster("local[*]").setAppName("MovieSimilarities") 
sc = SparkContext(conf = conf) 

We're doing this with a local[*] master argument here, which means we're actually going to use Spark's built-in cluster manager and treat every core on your desktop as a node on a cluster. For the first time, we're going to be running ...