Frank Kane's Taming Big Data with Apache Spark and Python

Loading the data

Now we're going to load up our data file. If you remember from Chapter 1, Getting Started With Spark, a very common way of creating an RDD is through the sc.textFile method. So, as you can see in the line of code shown here, this is actually going to go out to our local file system, go to the ml-100k rating dataset from MovieLens, and then load up the data file that includes all of the movie ratings data:

lines = sc.textFile("file:///SparkCourse/ml-100k/u.data") 

Now if you were to open up the u.data file in some sort of a text editor, it would look something like the following information, only with a hundred thousand lines:

What textFile does is it breaks up that input file line by line, so that every line of text corresponds ...