O'Reilly logo

Scala Data Analysis Cookbook by Arun Manivannan

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Creating a DataFrame from Scala case classes

In this recipe, we'll see how to create a new DataFrame from Scala case classes.

How to do it...

  1. We create a new entity called Employee with the id and name fields, like this:
    case class Employee(id:Int, name:String)
    

    Similar to the previous recipe, we create SparkContext and SQLContext.

    val conf = new SparkConf().setAppName("colRowDataFrame").setMaster("local[2]")
    
    //Initialize Spark context with Spark configuration.  This is the core entry point to do anything with Spark
    val sc = new SparkContext(conf) ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required