High Performance Spark, 2nd Edition

Chapter 6. Joins (SQL and Core)

Joining data is an important part of many of our pipelines, and both Spark Core and SQL support the same fundamental types of joins. They do, however, have very different execution options and performance. While joins are very common and powerful, they warrant special performance consideration, as they may require large network transfers or even create datasets beyond our capability to handle.¹ In core Spark, it can be more important to think about the ordering of operations, since the DAG optimizer, unlike the SQL optimizer, isn’t able to reorder or push down filters.

To understand the performance of joins in Spark, it’s important to distinguish two similar-sounding concepts: join types and join execution techniques. Join types impact the result and are things like left and right inner joins, outer joins, full self joins, etc. Join execution techniques refer to how the engine, Spark, computes the result and can be things like broadcast, hash, sort, etc. The join type, as well as the data sizes, conditions, configuration, and hints, all influence what join technique Spark chooses. Another way of thinking of this is the join type is the logical operation, and the join execution is the method Spark uses to implement a join.

Core Spark Joins

In this section, we will go over the RDD-type joins. Joins, in general, are expensive since they require that corresponding keys from each RDD be located on the same machine so that they can be combined locally. ...