O'Reilly logo

Mastering Apache Spark 2.x - Second Edition by Romeo Kienzler

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Hyperparameter tuning

CrossValidation is often used in conjunction with so-called (hyper)parameter tuning. What are hyperparameters? These are the various knobs that you can tweak on your machine learning algorithm. For example, these are some parameters of the Random Forest classifier:

  • Number of trees
  • Feature subset strategy
  • Impurity
  • Maximal number of bins
  • Maximal tree depth

Setting these parameters can have a significant influence on the performance of the trained classifier. Often, there is no way of choosing them based on a clear recipe--of course, experience helps--but hyperparameter tuning is considered as black magic. Can't we just choose many different parameters and test the prediction performance? Of course we can. This feature ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required