Scala and Spark for Big Data Analytics

Data manipulation

Show the column names in the SparkDataFrame as follows:

columns(flightDF)[1] "year" "month" "day" "dep_time" "dep_delay" "arr_time" "arr_delay" "carrier" "tailnum" "flight" "origin" "dest" [13] "air_time" "distance" "hour" "minute" 

Show the number of rows in the SparkDataFrame as follows:

count(flightDF)[1] 336776

Filter flights data whose destination is only Miami and show the first six entries as follows:

 showDF(flightDF[flightDF$dest == "MIA", ], numRows = 10)

The output is as follows:

Figure 25: Flights with destination Miami only

Select specific columns. For example, let's select all the flights that are going to Iowa ...