Apache Spark 2.x Cookbook

How to do it...

1. Start the Spark shell or the Databricks Cloud Scala notebook:

        $ spark-shell  

2. Create  the Person case class:

        scala> case class Person( first_name:String, last_name:String,           age:Int)

3. Load the person directory as a Dataset:

       scala> val p = spark.read.textFile("s3a://sparkcookbook/person")

4. Check the first item to kick the tires:

        scala> p.first

5. Split each line into an array of strings, based on a comma as the delimiter:

        scala> val pmap = p.map( line => line.split(","))

6. Convert the Dataset of Array[String] into the Dataset of Person case objects:

     scala> val personDS = pmap.map( p => Person(p(0),p(1),p(2).toInt))

7. Register personDS as a view:

        scala> personDS.createOrReplaceTempView("person")

8. Run a ...