Frank Kane's Taming Big Data with Apache Spark and Python

Finding the most popular movie

Let's start by reviewing the data format of the MovieLens dataset, the u.data file.

As you might recall, the u.data file on each line, consists of a user ID, a movie ID, a rating, and a timestamp. Each line says, "this user watched this movie, gave it this rating, and did it at this time":

Our task is to just figure out which movie was watched most often or which movie ID appears most frequently in the entire dataset. This isn't a very hard thing to do; in fact, if you want to go give it a crack yourself, feel free. In this section we'll take a look at the implementation that I came up with, get that to run, ...