Scala and Spark for Big Data Analytics

Working with DataFrames and RDDs

SparkDataFrame is a distributed collection of rows under named columns. Less technically, it can be considered as a table in a relational database with column headers. Furthermore, PySpark DataFrame is similar to Python pandas. However, it also shares some mutual characteristics with RDD:

• Immutable: Just like an RDD, once a DataFrame is created, it can't be changed. We can transform a DataFrame to an RDD and vice versa after applying transformations.
• Lazy Evaluations: Its nature is a lazy evaluation. In other words, a task is not executed until an action is performed.
• Distributed: Both the RDD and DataFrame are distributed in nature.

Just like Java/Scala's DataFrames, PySpark DataFrames are designed for ...