Frank Kane's Taming Big Data with Apache Spark and Python

Caching RDDs

I want to mention caching RDDs. I'll point this out as we go through the code in the next section. In our example, we're actually going to compute the RDD of movie similarities and query it a couple of times. The way Spark works is as soon as you call an action, it just discards that RDD, so if you actually want to use an RDD more than once and perform more than one action on it, you need to tell Spark explicitly to cache it and make it persistent. If you don't cache it, Spark might have to go back and reconstruct that RDD all over again and you might end up doubling the amount of work that you have to do. That would be bad because that self-join operation is very expensive in this example and we don't want to have to do that ...