Modern Scala Projects

Step 4 – applying the classifier to the test data

The purpose of our validation set is to be able to make a choice between models. We want an evaluation metric as well as hyperparameter tuning. Now, we will create an instance of a validation estimator called TrainValidationSplit, which will split the training set into a validation set and a training set as follows:

validatedTestResults.setEvaluator(new MulticlassClassificationEvaluator())

Next, we will fit this estimator over the training dataset to produce a model and a transformer that we will use to transform our testing dataset. Finally, we will perform validation for hyperparameter tuning by applying an evaluator for a metric.

The new ValidatedTestResults DataFrame should contain the ...