Scala Programming Projects

Transforming rows using map

Type the following in the Scala console:

val dsInt = dsString.map(_.toInt)// dsInt: org.apache.spark.sql.Dataset[Int] = [value: int]dsInt.explain()

You should see something similar to this:

== Physical Plan ==*(1) SerializeFromObject [input[0, int, false] AS value#96]+- *(1) MapElements <function1>, obj#95: int   +- *(1) DeserializeToObject value#91.toString, obj#94:     java.lang.String      +- LocalTableScan [value#91]

The explain() method shows the execution plan that will be run if we call an action method, such as show() or collect().

From this plan, we can deduce that calling map is not very efficient. Indeed, Spark stores the rows of Dataset off-heap in binary format. Whenever you call map, it has to deserialize this ...