High Performance Spark, 2nd Edition

Chapter 7. Effective Transformations

Most commonly, Spark programs are structured on RDDs and Datasets. Most Spark programs using the RDD APIs involve reading data from stable storage into the RDD format, performing a number of computations and data transformations on the RDD, and writing the resulting RDD to stable storage or collecting it to the driver. Even Dataset operations are largely compiled down into RDD transformations. Thus, most of the power of Spark comes from its transformations: operations that are defined on RDDs and return RDDs.

At present, Spark contains specialized functionality for about a half-dozen types of RDDs, each with its own properties and scores of different transformation functions. In this section, we hope to give you the tools to think about how your RDD transformation, or series of transformations, will be evaluated. In particular: what kinds of RDDs these transformations return, whether persisting or checkpointing RDDs between transformations will make your computation more efficient, and how a given series of transformations could be executed in the most performant way possible.

Note

The transformations in this section are those associated with the RDD object used in Spark Core. RDDs are also used inside of DStreams with legacy Spark Streaming, but they have different functionality and performance properties. Datasets, DataFrames, and Spark SQL (streaming or regular) all compile down to RDDs, with an additional optimizer in between. It is still ...