Machine Learning with Python Cookbook, 2nd Edition

Chapter 12. Model Selection

12.0 Introduction

In machine learning, we use training algorithms to learn the parameters of a model by minimizing some loss function. However, many learning algorithms (e.g., support vector classifier and random forests) have additional hyperparameters that are defined by the user and affect how the model will learn its parameters. As we mentioned earlier in the book, parameters (also sometimes called model weights) are what models learn during the training process, whereas hyperparameters are provided manually by us (the users).

For example, random forests are collections of decision trees (hence the word forest); however, the number of decision trees in the forest is not learned by the algorithm and must be set prior to fitting. This is often referred to as hyperparameter tuning, hyperparameter optimization, or model selection. Additionally, we might want to try multiple learning algorithms (for example, trying both support vector classifier and random forests to see which learning method produces the best model).

While there is widespread terminology variation in this area, in this book we refer to selecting both the best learning algorithm and its best hyperparameters as model selection. The reason is straightforward: imagine we have data and want to train a support vector classifier with 10 candidate hyperparameter values and a random forest classifier with 10 candidate hyperparameter values. The result is that we are trying to select the best ...