O'Reilly logo

Machine Learning with Spark - Second Edition by Nick Pentreath, Manpreet Singh Ghotra, Rajdeep Dua

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Iterations

As we saw when evaluating our classification models, we generally expect that a model trained with SGD will achieve better performance as the number of iterations increases, although the increase in performance will slow down as the number of iterations goes above some minimum number. Note that here, we will set the step size to 0.01 to better illustrate the impact at higher iteration numbers.

We implemented the same in Scala with different values of iterations, as follows:

val data = LinearRegressionUtil.getTrainTestData() val train_data = data._1 val test_data = data._2 val iterations = 10 //LinearRegressionCrossValidationStep$ //params = [1, 5, 10, 20, 50, 100, 200] val iterations_param = Array(1, 5, 10, 20, 50, 100, 200)  ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required