Deep Learning Quick Reference

Hyperband

Hyperband is a hyperparameter optimization technique that was developed at Berkley in 2016 by Lisha Li, Kevin Jamieson, Guilia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalker. You can read their original paper at https://arxiv.org/pdf/1603.06560.pdf.

Imagine randomly sampling many potential sets of hyperparameters, as we did above in RandomSearchCV. When RandomSearchCV is done, it will have chosen one single hyperparameter configuration as the best among those it sampled. Hyperband exploits the idea that a best hyperparameter configuration is likely to outperform other configurations after even a small number of iterations. The band in Hyperband comes from bandit, referring back to exploration versus exploitation based on multi-arm ...