Apache Spark Deep Learning Cookbook

There's more...

While we did do our joins using functions within a Spark dataframe using PySpark, we could have also done it by registering the dataframes as temporary tables and then joining them using sqlContext.sql():

1. First, we would register each of our datasets as temporary views using creatorReplaceTempView(), as seen in the following script:

movies.createOrReplaceTempView('movies_')links.createOrReplaceTempView('links_')ratings.createOrReplaceTempView('ratings_')

2. Next, we would write our SQL script just as we would do with any other relational database using the sqlContext.sql() function, as seen in the following script:

mainDF_SQL = \sqlContext.sql("""    select    r.userId_1    ,r.movieId_1    ,r.rating_1    ,m.title    ,m.genres    ,l.imdbId ,l.tmdbId ...