Scala and Spark for Big Data Analytics

Explicit schema

A schema is described using StructType, which is a collection of StructField objects.

StructType and StructField belong to the org.apache.spark.sql.types package. DataTypes such as IntegerType, StringType also belong to the org.apache.spark.sql.types package.

Using these imports, we can define a custom explicit schema.

First, import the necessary classes:

scala> import org.apache.spark.sql.types.{StructType, IntegerType, StringType}import org.apache.spark.sql.types.{StructType, IntegerType, StringType}

Define a schema with two columns/fields-an Integer followed by a String:

scala> val schema = new StructType().add("i", IntegerType).add("s", StringType)schema: org.apache.spark.sql.types.StructType = StructType(StructField(i,IntegerType,true), ...