Apache Spark 2.x Cookbook

How to do it...

1. Start the Spark shell or Scala notebook in the Databricks Cloud:

        $ spark-shell  

2. Create a case class for Person:

        scala> case class Person(firstName: String, lastName: String,           age:Int)

3. Load the person data as a DataSet and map it to the Person case class:

        scala> val personDS =          spark.read.textFile("s3a://sparkcookbook/person").map( line =>           line.split(",")).map( p => Person(p(0),p(1),p(2).toInt))

4. Register the person DataSet as a temp table so that SQL queries can be run against it. Note that the DataFrame name does not have to be the same as the table name:

        scala> person.createOrReplaceTempView("person")

5. Select all persons whose age is over 60 years:

        scala> val sixtyPlus = spark.sql("select * from person ...