Hands-On Big Data Analytics with PySpark

Leveraging JSON as a data format

In this section, we will leverage JSON as a data format and save our data in JSON. The following topics will be covered:

• Saving data in JSON format
• Loading JSON data
• Testing

This data is human-readable and gives us more meaning than simple plain text because it carries some schema information, such as a field name. We will then learn how to save data in JSON format and load our JSON data.

We will first create a DataFrame of UserTransaction("a", 100) and UserTransaction("b", 200), and use .toDF() to save the DataFrame API:

val rdd = spark.sparkContext         .makeRDD(List(UserTransaction("a", 100), UserTransaction("b", 200)))         .toDF()

We will then issue coalesce() and, this time, we will take the value as 2, and ...