June 2026
Intermediate to advanced
412 pages
10h 46m
English
Content preview from High Performance Spark, 2nd Edition
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
Appendix B. Spark Connect
Spark Connect decouples much of the driver from your Spark application. Spark Connect does not support the full Spark APIs, but if your application uses the restricted subset that is supported by Spark Connect, it gives you more flexibility to move between different backends. Since Spark Connect decouples the driver, you also have more flexibility with which language (and language version) you use to work with Spark.
With Spark Connect, a Spark application converts DataFrame/Dataset operations into protobuf messages and sends them to a remote Spark Connect server. This remote server performs many of the same tasks as the Spark driver does in the full version of Spark. Because the driver is already running separately, this can reduce the startup time for a Spark query, making Spark Connect especially well suited to smaller queries (like interactive analytics).
Warning
Some of the Spark Connect features discussed online, even in the Spark documentation, are not available in the open source “vanilla” Spark and only available from closed source vendors.
While Spark Connect gives you more flexibility to move between different execution engines, the limited API makes it poorly suited to many “high performance” applications where you want to access internals or exercise increased levels of control.
Supported API
Spark Connect supports most of the DataFrame/Dataset APIs and even has limited user-defined function support, but only in Python and Scala and only ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.