High Performance Spark, 2nd Edition

Chapter 4. What’s New in Spark 4.2 Since 2.4

Spark has changed significantly since 2.4, and one could fill an entire book of just the issues resolved. In this chapter, you will get an understanding of the most important performance-related changes in Spark that have occurred, as well as a bit of a look forward at some of the current proposals for large changes in the coming years.

Taking advantage of some of these changes can be as simple as upgrading to the latest version of Spark. Other changes require changing not just Spark but also other components in your ecosystem, like Iceberg or Delta Lake. Of course, some of the biggest improvements are in the form of new functionality, which you can use to either delete your own code or speed up existing tools.

It’s not all good news, though; there are places where functionality is being removed or no longer maintained. While Spark has a fairly comprehensive test suite so that even less-maintained components are unlikely to break, the performance may degrade between versions, as not every component has benchmarking.

The Bad News: GraphX

We decided to address the elephant in the room: Spark’s less-supported components. GraphX is much less well-maintained and could be considered functionally deprecated. In fact, there was a vote to deprecate it, which did pass (with some dissent), but it remains (technically) undeprecated as there is work being done to attempt to bring it back; we would be cautious before adopting it in any new project. ...