Frank Kane's Taming Big Data with Apache Spark and Python

Creating RDDs

One simple example of using sc is that you can use the sc.parallelize function to take a hardcoded set of data and make an RDD out of it. But that's not very interesting, it's not really going to be useful in a real production setting because if you could hardcode the data, then it's not really a big dataset to begin with now, is it? More often, we'll use something like sc.txtFile to create an RDD object. So, for example, if I have a giant text file full of, oh I don't know, movie ratings data for example, on my hard drive, we can use sc.txtFile to create an RDD object from SparkContext and then we can just use that RDD object going forward and process it:

sc.textFile("file:///c:/users/frank/gobs-o-text.txt") 

Now, again, if ...