Frank Kane's Taming Big Data with Apache Spark and Python

Examining the popular-movies script

In the download package for this book, you'll find a popular-movies Python script. Download that, put it in your SparkCourse folder, and open it up. We will print out a list of all of the movies and the number of times they appear, and then sort them based on how often they appear. It's a pretty simple script here, there's nothing really new in this example, but there will be when we build upon it:

from pyspark import SparkConf, SparkContextconf = SparkConf().setMaster("local").setAppName("PopularMovies")sc = SparkContext(conf = conf)lines = sc.textFile("file:///SparkCourse/ml-100k/u.data")movies = lines.map(lambda x: (int(x.split()[1]), 1))movieCounts = movies.reduceByKey(lambda x, y: x + y)flipped = ...