Learning PySpark

Creating DataFrames

Typically, you will create DataFrames by importing data using SparkSession (or calling spark in the PySpark shell).

Tip

In Spark 1.x versions, you typically had to use sqlContext.

In future chapters, we will discuss how to import data into your local file system, Hadoop Distributed File System (HDFS), or other cloud storage systems (for example, S3 or WASB). For this chapter, we will focus on generating your own DataFrame data directly within Spark or utilizing the data sources already available within Databricks Community Edition.

Note

For instructions on how to sign up for the Community Edition of Databricks, see the bonus chapter, Free Spark Cloud Offering.

First, instead of accessing the file system, we will create a DataFrame ...