Deep Learning Quick Reference

Common strategies

There are a common set of strategies for hyperparameter optimization that are used across all machine learning models. At a high level, those strategies include the following:

• Grid search
• Random search
• Bayesian optimization
• Genetic algorithms
• Machine learning for hyperparameters

Grid search is simply the act of trying everything, or at least discrete chunks of everything, and then reporting on the best combination of hyperparameters we've found with brute force. It's guaranteed to find the best solution across the parameter space we've identified, along with every other less good solution.

Grid search isn't very practical for deep learning though. We can't realistically explore every possible value of every possible parameter ...