Apache Spark Deep Learning Cookbook

How to do it...

This section walks through the steps to evaluate the model performance.

1. Create a confusion matrix using the .crosstab() function, as seen in the following script:

df_predicted.crosstab('label', 'prediction').show()

2. Import metrics from sklearn to help measure accuracy using the following script:

from sklearn import metrics

3. Create two variables for the actual and predicted columns from the dataframe that will be used to measure accuracy, using the following script:

actual = df_predicted.select('label').toPandas()predicted = df_predicted.select('prediction').toPandas()

4. Compute the accuracy prediction score using the following script:

metrics.accuracy_score(actual, predicted)