High Performance Spark, 2nd Edition

Chapter 5. DataFrames, Datasets, and Spark SQL

In 2017, we predicted that the future of Spark would be defined by Spark SQL DataFrames and Datasets interfaces’ simplicity and ease of adoption. Today they are the most used capabilities among Spark developers. With more efficient storage options, an advanced optimizer, native accelerators, and direct operations on serialized data, this has become the go-to distributed data processing capabilities for data engineers, data analysts, and data scientists.

These components are super important for getting the best of Spark performance (see Figure 5-1). Datasets are a Scala/Java-specific concept versus DataFrames, which are supported across languages, including PySpark. For the most part we’ll use Dataset instead of spelling out Dataset and DataFrame except for the section headers, where we’ve separated out some Dataset-specific information. When talking about things that apply to Spark SQL regardless of interface (SQL, DataFrame, or Dataset) we’ll just say Spark SQL.

Datasets, and their earlier untyped DataFrames, specify more of what the goal (declarative) is versus specifying how to compute the value (imperative) as in Spark RDDs. The declarative approach opens up even more optimizations on top of those opened up by Spark RDDs’ functional APIs with lazy evaluation. When possible, we generally recommend using Datasets in most instances instead of RDDs.

Figure 5-1. Relative performance for RDD versus DataFrames based on SimplePerfTest ...