Apache Spark 2.x Cookbook

There's more...

This recipe was about learning how to use case classes to attach schema and the input data we used here is in CSV format. Also, there is another way to load data:

scala> val people = spark.read.format("csv").load("s3a://sparkcookbook/person")

The output looks like this:

+------+-------+---+| _c0| _c1|_c2|+------+-------+---+|Barack| Obama| 55||George| Bush| 70|| Bill|Clinton| 70|+------+-------+---+

Now there are two ways to fix this. The first approach is to supply custom column names, such as the following:

scala> val people = spark.read.format("csv").load("s3a://sparkcookbook/person").select($"_c0".as("first_name"),$"_c1".as("last_name"),$"_c2".as("age").cast("Int"))

The second way is to load data that already has ...