Scala and Spark for Big Data Analytics

Getting started

Let's start with loading, parsing, and viewing simple flight data. At first, download the NY flights dataset as a CSV from https://s3-us-west-2.amazonaws.com/sparkr-data/nycflights13.csv. Now let's load and parse the dataset using read.csv() API of R:

#Creating R data framedataPath<- "C:/Exp/nycflights13.csv"df<- read.csv(file = dataPath, header = T, sep =",")

Now let's view the structure of the dataset using View() method of R as follows:

View(df)

Figure 20: A snap of the NYC flight dataset

Now let's create the Spark DataFrame from the R DataFrame as follows:

##Converting Spark DataFrame  flightDF<- as.DataFrame(df)

Let's ...