Scala and Spark for Big Data Analytics

Querying SparkR DataFrame

Similar to Scala, we can perform a SQL query on the DataFrame once it is saved as TempView using the createOrReplaceTempView() method. Let's see an example of that. At first, let's save the fight DataFrame (that is, flightDF) as follows:

# First, register the flights SparkDataFrame as a tablecreateOrReplaceTempView(flightDF, "flight")

Now let's select destination and destinations of all the flights with their associated carrier information as follows:

destDF<- sql("SELECT dest, origin, carrier FROM flight")  showDF(destDF, numRows=10)

The output is as follows:

Figure 29: All the flights with their associated carrier ...