O'Reilly logo

Effective Amazon Machine Learning by Alexis Perrier

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Multiplying trials

The evaluation scores on the various models and dataset version are, to a certain extent, dependent on the samples contained in the evaluation sets. If we run the following experiment several times on the three datasets, we see certain variations in the scores:

  • Shuffle and split the dataset into three -- training, validation, and held-out and create the respective datasources
  • Train a model on the training dataset, keeping the default Amazon ML settings (mild L2 regularization)
  • Evaluate the model on the evaluation and held-out datasets

The following plot shows the respective performances of the three models for several trials. The average AUC is written on the graph. We see that on average, the extended dataset performs ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required