Tidy Modeling with R

Chapter 12. Model Tuning and the Dangers of Overfitting

In order to use a model for prediction, the parameters for that model must be estimated. Some of these parameters can be estimated directly from the training data, but other parameters, called tuning parameters or hyperparameters, must be specified ahead of time and can’t be directly found from training data. These are unknown structural values or other kinds of values that have significant impact on the model but cannot be directly estimated from the data. This chapter will provide examples of tuning parameters and show how we use the tidymodels functions to create and handle tuning parameters. We’ll also demonstrate how poor choices of these values lead to overfitting and introduce several tactics for finding optimal tuning parameters values. Chapters 13 and 14 go into more detail on specific optimization methods for tuning.

Model Parameters

In ordinary linear regression, there are two parameters ${\beta }_0$ and ${\beta }_1$ of the model:

$$y_i={\beta }_0+{\beta }_1{x}_i+{ϵ}_i$$

When we have the outcome ($y$) and predictor ($x$) data, we can estimate the two parameters ${\beta }_0$ and ${\beta }_1$:

$${\widehat{\beta }}_1=\frac{{\sum }_i(y_i-\overline{y})(x_i-\overline{x})}{{\sum }_i{(x_i-\overline{x})}^2}$$

and

$${\widehat{\beta }}_0=\overline{y}-{\widehat{\beta }}_1\overline{x}.$$

We can directly estimate these values from the data for this example model because they are analytically tractable; if we have the data, then we can estimate these model parameters.

Note

There are many situations where a model has parameters that can’t be ...