Apache Spark 2.x Cookbook

How to do it...

1. Start the Spark shell or Databricks Cloud Scala notebook:

        $ spark-shell  

2. Import the Spark SQL datatypes and Row objects:

        scala> import org.apache.spark.sql._        scala> import org.apache.spark.sql.types._

3. Create the schema using the StructType and StructField objects. The StructField object takes parameters in the form of param name, param type, and nullability:

        scala> val schema = StructType(    Array(StructField("first_name",StringType,true),StructField("last_name",StringType,true),StructField("age",IntegerType,true)))

4. Load the person data in a Dataset and attach the schema to it:

        scala> val personDS =         spark.read.schema(schema).csv("s3a://sparkcookbook/person")

5. Register personDS as a view:

        scala> personDS.createOrReplaceTempView("person") ...