book

High Performance Spark, 2nd Edition

by Holden Karau, Adi Polak, Rachel Warren

June 2026

Intermediate to advanced

412 pages

10h 46m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Preface
Second Edition NotesSupporting Books and MaterialsConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsGeneral AcknowledgmentsSecond EditionFirst EditionPersonal AcknowledgmentsHolden KarauAdi Polak
1. Introduction to High Performance Spark
What Is Spark and Why Performance MattersWhat You Can Expect to Get from This BookSpark VersionsWhy the Focus on Scala and Python?To Be a Spark Expert You Have to Be Able to Read a Little Scala AnywayThe Spark Scala and Python APIs Are Easier to Use Than the Java APIsWhy Not Scala?Learning ScalaConclusion
2. How Spark Works
How Spark Fits into the Big Data EcosystemCluster ManagersCatalog SystemMultitool PipelinesSpark ComponentsSpark Model of Parallel Computing: RDDsLazy EvaluationIn-Memory Persistence and Memory ManagementImmutability and the RDD InterfaceTypes of RDDsFunctions on RDDs: Transformations Versus ActionsWide Versus Narrow DependenciesSpark Job SchedulingResource AllocationThe Spark ApplicationDefault Spark Scheduler and Concurrent ActionsThe Anatomy of a Spark JobThe DAGJobsStagesTasksSpark ConnectConclusion
3. Upgrading Spark
Finding What You Need to ChangeCompile-Time ChangesExceptions at RuntimeDiffering ResultsUpdating Your CodeScalaPythonSQLVerifying Correctness and PerformancePlaces Where Performance Can Get WorseConclusion
4. What’s New in Spark 4.2 Since 2.4
The Bad News: GraphXThe Good NewsNo Code Changes RequiredPartition Pruning and Predicate PushdownJoinsDebuggingAdditional Upgrades Required (Not Code)Code and Logic Changes NeededGeneralPython SpecificScala SpecificPrioritizing UpgradesTotally New FeaturesSpark Declarative PipelinesSQL UDFs/UDTFsLooking Ahead to the FutureConclusion
5. DataFrames, Datasets, and Spark SQL
Getting Started with the SparkSession (or HiveContext or SQLContext)Spark SQL DependenciesBasics of SchemasColumn MetadataDataFrame APITransformationsMulti-DataFrame TransformationsPlain Old SQL Queries and Interacting with Metastore Tables (Hive, Iceberg, etc.)Data Representation in DataFrames and DatasetsTungstenDatasetsInteroperability with RDDs, DataFrames, and Local CollectionsCompile-Time Strong TypingEasier Functional (RDD “like”) TransformationsRelational TransformationsMulti-Dataset Relational TransformationsGrouped Operations on DatasetsExtending with User-Defined Functions (UDFs), Aggregate Functions (UDAFs), and ExpressionsCustom ExpressionsCustom EncodersUser-Defined Table FunctionsData Loading and Saving FunctionsDataFrameWriter and DataFrameReaderFormatsSave ModesMergePartitions (Discovery and Writing)Query OptimizerLogical and Physical PlansStatic Optimizer Rules You Should Understand (and Working Around Them)Understanding Spark Adaptive Query ExecutionCode GenerationLarge Query Plans and Iterative AlgorithmsSparkSession Extensions and Custom OptimizersDebugging Spark SQL QueriesJDBC/ODBC ServerConclusion
6. Joins (SQL and Core)
Core Spark JoinsChoosing a Join TypeChoosing an Execution PlanSpark SQL JoinsDataFrame JoinsConcrete SQL Join Execution OperatorsDataset JoinsConclusion
7. Effective Transformations
Narrow Versus Wide TransformationsImplications for PerformanceImplications for Fault ToleranceThe Special Case of coalesceWhat Type of RDD Does Your Transformation Return?Minimizing Object CreationReusing Existing ObjectsUsing Smaller Data StructuresIterator-to-Iterator Transformations with mapPartitionsWhat Is an Iterator-to-Iterator Transformation?Space and Time AdvantagesAn ExampleSet OperationsReducing Setup OverheadShared VariablesBroadcast VariablesAccumulatorsReusing RDDsCases for ReuseDeciding if Recompute Is Inexpensive EnoughTypes of Reuse: Cache, Persist, Checkpoint, Shuffle FilesAlluxio (nee Tachyon)LRU CachingShuffle FilesNoisy Cluster ConsiderationsInteraction with AccumulatorsConclusion
8. Working with Key/Value Data
The Goldilocks ExampleGoldilocks Version 0: Iterative SolutionHow to Use PairRDDFunctions and OrderedRDDFunctionsActions on Key/Value PairsWhat’s So Dangerous About the groupByKey Function?Goldilocks Version 1: groupByKey SolutionChoosing an Aggregation OperationDictionary of Aggregation Operations with Performance ConsiderationsPreventing Out-Of-Memory Errors with Aggregation OperationsMultiple RDD OperationsCo-GroupingPartitioners and Key/Value DataUsing the Spark Partitioner ObjectHash PartitioningRange PartitioningCustom PartitioningPreserving Partitioning Information Across TransformationsLeveraging Co-Located and Co-Partitioned RDDsDictionary of Mapping and Partitioning Functions PairRDDFunctionsDictionary of OrderedRDDOperationsSorting by Two Keys with sortByKeySecondary Sort and repartitionAndSortWithinPartitionsLeveraging repartitionAndSortWithinPartitions for a Group by Key and Sort Values FunctionHow Not to Sort by Two OrderingsGoldilocks Version 2: Secondary SortA Different Approach to GoldilocksGoldilocks Version 3: Sort on Cell ValuesStraggler Detection and Unbalanced DataBack to Goldilocks (Again)Goldilocks Version 4: Reduce to Distinct on Each PartitionConclusion
9. Going Beyond Scala
Beyond Scala Within the JVMCustom Code Beyond Scala and Beyond the JVMHow SparklyR WorksSpark on the Common Language Runtime (CLR)—C# and FriendsCalling Other Languages from SparkUsing Pipe and FriendsJava Native InterfaceJava Native Access (JNA)Project PanamaUnderneath Everything Is FortranGetting to the GPUGoing Beyond the JVM with Spark AcceleratorsDatabricks PhotonApache Arrow DataFusion CometProject GlutenSpark RAPIDSApplication-Specific Integrated Circuits (ASICs)Managing Memory Outside of the JVMThe Future (from ~2026)Conclusion

10. Spark: A Thoughtful Step into Generative AI
So, What Is This Generative AI Thing?How Generative AI Works: Training the CreatorA World of Applications: Generative AI at WorkHow Is GenAI Related to Deep Learning?Can You Spark It?Effortless TensorFlow Scaling with Spark’s TensorFlowDistributorScaling PyTorch with Spark’s TorchDistributorSupercharging Distributed Training with DeepSpeedWait, Your Processor Matters!Inference in the Real World: Beyond Batch PredictionBatch Prediction: Scaling Predictions with SparkModel as a Service: Real-Time Predictions at ScaleModel in a Service: Embedding Inference into WorkflowsA World of Inference PossibilitiesEvaluation of ModelsCase StudiesFight Health InsuranceUber’s Enhanced Agentic-RAG (EAg-RAG): When Chatbots Deliver Near-Human PrecisionThe Model Dilemma: Hosting or Leveraging a Third-Party REST APILayer 1: Use Case and Domain AlignmentLayer 2: Performance RequirementsLayer 3: Scalability NeedsLayer 4: Cost ConsiderationsLayer 5: Data Sensitivity and SecurityLayer 6: Integration and Ecosystem FitLayer 7: Time-to-MarketThe DecisionAlex’s Reflection MomentThe Story of an Evolving IndustryConclusion: The Future of Generative AI with SparkSpark’s Role in the Generative AI RevolutionGenerative AI and the Creative IndustriesThe Path ForwardYour Next Step
11. Testing, Validation, and Side-By-Side Runs
Unit TestingFactoring Your Code for TestabilityMocking RDDsCore Spark Jobs (Testing with RDDs)Testing StreamingTesting DataFramesTesting CodegenGetting Test DataGenerating Large DatasetsSamplingProperty Checking with ScalaCheckComputing RDD DifferenceIntegration TestingChoosing Your Integration Testing EnvironmentVerifying PerformanceSpark Counters for Verifying PerformanceProjects for Verifying PerformanceValidation (or Audits)Data ValidationAccumulators and Built-In CountersSide-by-Side RunsTable-Level AssertionsConclusion
12. Spark Components and Packages
GraphXUsing Community Packages and LibrariesPopular PackagesCreating a Spark PackageBuilding Your Own Spark or Using PackagesConclusion
A. Just Enough Iceberg and Friends
Updates and Table MaintenancePartitioning, Indices, and StatisticsQuery PlanningConclusion
B. Spark Connect
Supported APIClient LanguagesDifferent BackendsConclusion
C. When Not to Use Spark
Small Data and Small TasksReal-Time Updates and Online Transaction ProcessingUser-Facing Interactive Use CasesLong and Highly Variable Task Processing TimeUntrusted and Multitenant QueriesYour Job Has Side Effects per RecordHuge Individual Records/RowsNo Ops Support and No Budget for a PlatformHighly Unreliable BackendsDifferent Functions Versus Same Function on Different DataFrequently Accumulated UpdatesGlobal In-Order Processing RequiredWhen You Can, Now, Use SparkConclusion
D. Advanced Task Scheduling: Gangs and Resource Profiles
Gang Scheduling/Barrier Execution ModeResource ProfilesConclusion
E. Spark Streaming
Simplest ExampleDriver ConsiderationsOperation ModesMicro-BatchingReal-Time ModeAsync Progress TrackingOutput ModesTime, Aggregations, Joins, and StateAggregationsStream-Stream JoinsReliability: YOLO-ish, At Least Once, At Most Once, and Exactly OnceCheckpointingReplayable and Nonreplayable ReceiversIdempotent and Non-Idempotent SinksTrue “Real-Time” and Reasonable ExpectationsConclusion
F. The Spark Web UI: Debugging and Optimizing Your Jobs
A Real-World Performance ProblemAccessing the Spark Web UIUnderstanding the Spark Execution HierarchyThe Jobs Tab: Our First Look at the ProblemThe Stages Tab: Finding the BottleneckThe SQL Tab: Understanding Query PlansThe Executors Tab: Resource UtilizationThe Problem with the Native UIEnter the DataFlint TabThe Fix: Actionable RecommendationsOther DataFlint Performance AlertsThe Spark History Server
Index
About the Authors

Content preview from High Performance Spark, 2nd Edition

About the Authors

Holden Karau is a queer transgender Canadian, Apache Spark committer, Apache Software Foundation member, and an active open source contributor. As a software engineer, she’s worked on a variety of distributed computing, search, AI, and classification problems at Apple, Netflix, Google, IBM, Alpine, Databricks, Foursquare, and Amazon. She graduated from the University of Waterloo with a bachelor of mathematics in computer science. In addition to her big data work, she cofounded Fight Health Insurance to help patients appeal health insurance denials. Outside of software, she enjoys playing with fire, welding, riding motorcycles, eating poutine, and dancing.

For most of Adi Polak’s professional life, she dealt with data and machine learning. As a data practitioner, she developed algorithms to solve real-world problems using machine-learning techniques. As an engineer, she brought her hands-on machine learning experience to various Fortune 500 companies’ products and services by building upon cutting-edge and emerging technologies. Adi has been working and contributing to the Apache Spark community since 2013 and taught Spark to thousands of students throughout the year. Adi is an official Databricks ambassador, the author of the successful book, Scaling Machine Learning with Spark (O’Reilly), and a respected worldwide presenter.

Rachel Warren is a data scientist and software engineer at Alpine Data Labs, where she uses Spark to address real-world data processing ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781098145842Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

High Performance Spark, 2nd Edition

by Holden Karau, Adi Polak, Rachel Warren

About the Authors

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.