Frank Kane's Taming Big Data with Apache Spark and Python

Preparing the script

Open up the movie-similarities-1m.py script:

As you can see, to make life easier, I actually went ahead and copied the ml-1m 1 million rating dataset to a sundog-spark bucket on Amazon's S3 service, so that my Spark cluster can access it:

#aws s3 cp s3://sundog-spark/m1-1m/movies.dat ./ 

I've also copied the script itself to my sundog-spark S3 bucket as well. This way I can quickly obtain a copy of it from my master node on my cluster:

#aws s3 cp s3://sundog-spark/MovieSimilarities1M.py ./ 

Once I spin up that cluster, the first thing I'm going to do, because time is money and the clock starts ticking right away, is ...