Frank Kane's Taming Big Data with Apache Spark and Python

Using .partitionBy()

The partitionBy() method is on an RDD that you can use to say, "I'm going to run some large operation and I know I have enough compute resources to actually split this up into many different executors, many different runs", This tells you how many pieces we want to break this job up into. If we call partitionBy() on the RDD that we do a self-join operation on first, we can then split it up into smaller chunks that a given executor can actually deal with. Then our job will complete successfully. Once you do that, Spark will attempt to preserve that partitioning going forward.

I want you to take a look at this list of different RDD methods:

• Join()
• cogroup()
• groupWith()
• join()
• leftOuterJoin()
• rightOuterJoin()
• groupByKey() ...