Apache Spark 2.x Cookbook

How it works...

A StructType object defines the schema. You can consider it equivalent to a table or a row in the relational world. StructType takes in an array of the StructField objects, as in the following signature:

StructType(fields: Array[StructField])

A StructField object has the following signature:

StructField(name: String, dataType: DataType, nullable: Boolean = true, metadata: Metadata = Metadata.empty)

Here is some more information on the parameters used:

• name: This represents the name of the field.
• dataType: This shows the datatype of the field.
•  The following datatypes are allowed:
  • IntegerType
  • FloatType
  • BooleanType
  • ShortType
  • LongType
  • ByteType
  • DoubleType
  • StringType

• nullable: This shows whether the field can be null. ...