Apache Spark Deep Learning Cookbook

How to do it...

This section will walk through the following steps for joining dataframes in PySpark:

1. Execute the following script to rename all field names in ratings, by appending a _1 to the end of the name:

for i in ratings.columns:     ratings = ratings.withColumnRenamed(i, i+'_1') 

2. Execute the following script to inner join the movies dataset to the ratings dataset, creating a new table called temp1:

temp1 = ratings.join(movies, ratings.movieId_1 == movies.movieId, how = 'inner')

3. Execute the following script to inner join the temp1 dataset to the links dataset, creating a new table called temp2:

temp2 = temp1.join(links, temp1.movieId_1 == links.movieId, how = 'inner')

4. Create our final combined dataset, mainDF, by left-joining