Apache Spark Deep Learning Cookbook

How to do it...

This section walks through the steps to evaluate the TF-IDF NLP model.

1. Create a confusion matrix using the following script:

predictionDF.crosstab('label', 'prediction').show()

2. Evaluate the model using metrics from sklearn with the following script:

from sklearn import metricsactual = predictionDF.select('label').toPandas()predicted = predictionDF.select('prediction').toPandas()print('accuracy score: {}%'.format(round(metrics.accuracy_score(actual,         predicted),3)*100))

3. Calculate the ROC score using the following script:

from pyspark.ml.evaluation import BinaryClassificationEvaluatorscores = predictionDF.select('label', 'rawPrediction')evaluator = BinaryClassificationEvaluator()print('The ROC score is {}%'.format(round(evaluator.evaluate(scores),3)*100)) ...