Frank Kane's Taming Big Data with Apache Spark and Python

Using SparkSQL in Python

In Python, it's pretty straightforward to use structured data. There are actually a couple of different interfaces that I'm going to look at in more detail in the examples to come. In short, you can do any of the following:

• Import things directly from a JSON file
• Import things directly from a Hive database
• Connect directly to a JDBC database
• Convert RDDs into DataFrames if you know their schema under the hood

Here's a quick example to showcase some of those imports:

from pyspark.sql import SparkSession, Row 
hiveContext = HiveContext(sc) 
inputData = spark.read.json(dataFile) 

Once you have a DataFrame, you can do things such as create or replace TempView to create a table in memory that you can issue SQL queries ...