O'Reilly logo

Effective Amazon Machine Learning by Alexis Perrier

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Evaluating the held-out dataset

In Chapter 5, Model Creation, we evaluated the performance of our different models on a slice of the training datasource. We obtained for each model an AUC score, and selected the AUC with the best AUC score. We relied on Amazon ML to create the validation set, by splitting the training dataset into two, with 70% for training and 30% of the data for validation. We could have done that split ourselves, created the validation datasource, and specified which datasource to use for the evaluation of the model.

In fact, nothing prevents us from running a model evaluation on the held-out dataset. If you go to the model summary page, you will notice a Perform another Evaluation button in the Evaluation section:

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required