Apache Spark Deep Learning Cookbook

How to do it...

This section will walk through the steps to analyze the movie ratings in the MovieLens database:

1. Retrieve some summary statistics on the rating_1 column by executing the following script:

mainDF.describe('rating_1').show

2. Build a histogram of the distribution of ratings by executing the following script:

import matplotlib.pyplot as plt%matplotlib inlinemainDF.select('rating_1').toPandas().hist(figsize=(16, 6), grid=True)plt.title('Histogram of Ratings')plt.show()

3. Execute the following script to view the values of the histogram in a spreadsheet dataframe:

mainDF.groupBy(['rating_1']).agg({'rating_1':'count'})\ .withColumnRenamed('count(rating_1)', 'Row Count').orderBy(["Row Count"],ascending=False)\ .show()

4. A unique count ...