Spark model metrics

At first, use the embedded model metrics which the Spark API provides. We are going to use the same approach that we used in the previous chapter. We start by defining a method to extract model metrics for a given model and dataset:

import org.apache.spark.mllib.evaluation._ 
import org.apache.spark.mllib.tree.model._ 
def getMetrics(model: RandomForestModel, data: RDD[LabeledPoint]): 
    MulticlassMetrics = { 
        val predictionsAndLabels = => 
            (model.predict(example.features), example.label) 
        new MulticlassMetrics(predictionsAndLabels) 

Then we can directly compute Spark MulticlassMetrics:

val rfModelMetrics = getMetrics(rfModel, testData) 

And look at first interesting classification model metrics called Confusion ...

Get Mastering Machine Learning with Spark 2.x now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.