Scala and Spark for Big Data Analytics

Pivots

Pivoting is a great way of transforming the table to create a different view, more suitable to doing many summarizations and aggregations. This is accomplished by taking the values of a column and making each of the values an actual column.

To understand this better, let's pivot the rows of the DataFrame by Year and examine the result, which shows that, now, the column Year created several new columns by converting each unique value into an actual column. The end result of this is that, now, instead of just looking at year columns, we can use the per year columns created to summarize and aggregate by Year.

scala> statesDF.groupBy("State").pivot("Year").sum("Population").show(5)+---------+--------+--------+--------+--------+--------+--------+--------+ ...