Hands-On Automated Machine Learning

Hyperparameter Optimization

The auto-sklearn library uses Bayesian optimization to tune the hyperparameters of machine learning (ML) pipelines. You will learn the inner workings of Bayesian optimization, but let's first review the basics of mathematical optimization.

In simple terms, optimization deals with selecting the best values to minimize or maximize a given function. A function is called a loss function or a cost function if our objective is minimization. If you are trying to maximize it, then it's called a utility function or a fitness function. For example, when you are building ML models, a loss function helps you to minimize the prediction error during the training phase.

When you look at this whole process from a wider angle, ...