High Performance Spark, 2nd Edition

Chapter 3. Joins (SQL and Core)

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 6th chapter of the final book. Please note that the GitHub repo is available at https://github.com/high-performance-spark/high-performance-spark-examples.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at [e-mail].

Joining data is an important part of many of our pipelines, and both Spark Core and SQL support the same fundamental types of joins. While Core and SQL support the same types they have very different execution options and performance. While joins are very common and powerful, they warrant special performance consideration as they may require large network transfers or even create datasets beyond our capability to handle.1 In core Spark it can be more important to think about the ordering of operations, since the DAG optimizer, unlike the SQL optimizer, isn’t able to re-order or push down filters.

To understand the performance of joins in Spark it’s important to distinguish two similar-sounding concepts, join types and join execution techniques. Join types impact the result and are things like left and right inner joins, outer joins, full-self-joins, ...