Scala Programming Projects

Transforming rows using select

A more efficient way to transform rows is to use select(cols: Column*):

import org.apache.spark.sql.DataFrameimport org.apache.spark.sql.types.IntegerTypeval df = ds.select($"value".cast(IntegerType))// df: org.apache.spark.sql.DataFrame = [value: int]val dsInt = df.as[Int]// dsInt: org.apache.spark.sql.Dataset[Int] = [value: int]

The implicits that we imported earlier let us use the $"columnName" notation to produce a Column object from String. The string after the $ sign must refer to a column that exists in the DataFrame source; otherwise, you would get an exception.

We then call the .cast method to transform each String into Int. But, at this stage, the resulting df object is not of the Dataset[Int]; type; ...