book

High Performance Spark, 2nd Edition

by Holden Karau, Adi Polak, Rachel Warren

May 2026

Intermediate to advanced

350 pages

2h 50m

English

O'Reilly Media, Inc.

Read now

Unlock full access

What Is Spark and Why Performance MattersWhat You Can Expect to Get from This BookSpark VersionsWhy the focus on Scala and Python?The Spark Scala and Python APIs Are Easier to Use Than the Java APIWhy Not Scala?Learning ScalaConclusion
Finding What You Need to ChangeCompile Time ChangesExceptions at RuntimeDiffering ResultsUpdating Your CodeScalaPythonSQLVerifying Correctness and PerformanceConclusion
Core Spark JoinsChoosing a Join TypeChoosing an Execution PlanSpark SQL JoinsDataFrame JoinsConcrete SQL Join Execution OperatorsUsing Join HintsDataset JoinsConclusion
Beyond Scala within the JVMCustom Code Beyond Scala, and Beyond the JVMHow PySpark WorksHow SparkR WorksSpark on the Common Language Runtime (CLR)—C# and FriendsCalling Other Languages from SparkUsing Pipe and FriendsJNIJava Native Access (JNA)Project PanamaUnderneath Everything Is FORTRANGetting to the GPUGoing Beyond the JVM with Spark AcceleratorsDatabricks PhotonApache Arrow Comet DatafusionProject GlutenSpark RAPIDsApplication-Specific Integrated Circuits (ASICS)Managing Memory Outside of the JVMThe future (from ~2024)Conclusion
Unit TestingFactoring your code for testabilityMocking RDDsCore Spark jobs (testing with RDDs)DStream StreamingTesting DataFramesTesting CodegenGetting Test DataGenerating Large DatasetsSamplingProperty Checking with ScalaCheckComputing RDD DifferenceIntegration TestingChoosing Your Integration Testing EnvironmentVerifying PerformanceSpark Counters for Verifying PerformanceProjects for Verifying PerformanceValidation (or Audits)Data ValidationCounters Built-in and AccumulatorsSide by Side RunsConclusion
GraphXUsing Community Packages and LibrariesPopular PackagesCreating a Spark PackageBuilding your own Spark v.s. PackagesConclusion

Content preview from High Performance Spark, 2nd Edition

Chapter 1. Introduction to High Performance Spark

This chapter provides an overview of what we hope you will be able to learn from this book and does its best to convince you to learn to read some Scala and consider writing your Spark jobs in Scala or Python.

Feel free to skip ahead to Chapter 2 if you already know what you’re looking for.

What Is Spark and Why Performance Matters

ASF (currently) stands for Apache Software Foundation, although there are calls to rename the foundation. Spark is a high-performance, general-purpose distributed computing system that has become the most active ASF open source project, with more than 1,000 active contributors.¹

Spark enables us to process large quantities of data, beyond what can fit on a single machine, with a high-level, relatively easy-to-use API. Spark’s design and interface are unique, and it is one of the fastest systems ...