Apache Spark Quick Start Guide

Encoders

Encoders are required to map domain-specific objects of type T to Spark's type system or internal Spark SQL representation. An Encoder of type T is a trait represented by Encoder[T].

Encoders are available with every Spark session, and you can explicitly import them with spark implicits as import spark.implicits._.

For example, given an Employee class with the fields name (String) and salary (int), an encoder is used as an indicator to serialize the Employee object to binary form. This binary structure provides the following advantages:

• Occupies less memory
• Data is stored in columnar format for efficient processing

Let's take a look at the major encoder features:

• Fast serialization: Encoders are used for runtime code with custom ...