Frank Kane's Taming Big Data with Apache Spark and Python

Differences between DataFrames and DataSets

Until Spark 2.0, people talked about DataFrames a lot, and the term made its way into the lexicon of Spark. You'd hear people talking about DataFrame APIs. However, in Spark 2.0, a DataFrame is a DataSet of Row objects. So now, in Spark 2.0, they talk about DataSets, and this can get a little bit confusing. A DataSet is just a DataFrame of structured data, so it can contain more than one Row object; it can contain a specific type of class. This is more important if you're coding in Scala or Java. Since Python is untyped to begin with, you're not really going to sense the difference when you're coding Spark code in Python. You just need to be aware of the terminology. Therefore, when people talk ...