Machine Learning with Spark - Second Edition

Movie dataset

Next, we will investigate a few properties of the movie catalog. We can inspect a row of the movie data file, as we did for the user data earlier, and then count the number of movies:

We will create a DataFrame of movie data by parsing using the format com.databrick.spark.csv and giving a | delimiter. Then, we use a CustomSchema to populate the DataFrame and return it:

def getMovieDataDF() : DataFrame = {   val customSchema = StructType(Array(   StructField("id", StringType, true),   StructField("name", StringType, true),   StructField("date", StringType, true),   StructField("url", StringType, true)));   val movieDf = spark.read.format(    "com.databricks.spark.csv")      .option("delimiter", "|").schema(customSchema)  .load(PATH_MOVIES) ...