O'Reilly logo

Machine Learning with Spark - Second Edition by Nick Pentreath, Manpreet Singh Ghotra, Rajdeep Dua

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Iterations

Many machine learning methods are iterative in nature, converging to a solution (the optimal weight vector that minimizes the chosen loss function) over a number of iteration steps. SGD typically requires relatively few iterations to converge to a reasonable solution, but can be run for more iterations to improve the solution. We can see this by trying a few different settings for the numIterations parameter, and comparing the AUC results like this:

val iterResults = Seq(1, 5, 10, 50).map { param =>   val model = trainWithParams(scaledDataCats, 0.0, param, new  SimpleUpdater, 1.0)   createMetrics(s"$param iterations", scaledDataCats, model) } iterResults.foreach { case (param, auc) => println(f"$param, AUC =  ${auc * 100}%2.2f%%") ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required