Decision Theory

11Choosing models

In this chapter we discuss model choice. So far we postulated a fixed data-generating mechanism f(x|θ) without worrying about how f is chosen. From this perspective, we may think about model choice as the choice of which f to use. Depending on the application, f could be a simple parametric family, or a more elaborate model. George Box’s famous comment that “all models are wrong but some models are useful” (Box 1979) highlights the importance of taking a pragmatic viewpoint in evaluating models, and to set criteria driven by the goals of the modeling. Decision theory would seem to be the perfect perspective to formalize Box’s concise statement of principle.

A view we could take is that of Chapter 10. Forecasters are incarnations of predictive models that we can evaluate and compare based on utility functions such as scoring rules. If we do this, we neatly separate the information that was used by the forecasters to develop and tune the prediction models, from the information that we use to evaluate them. This separation is a luxury we do not always have. More often we would like to be able to entertain several approaches in parallel, and learn something about how well they do directly from the data that are used to develop them. Whether this is even possible is a matter of debate, and some hold, with good reasons, that model training and model assessment should be separate.

But let us say we give in to the temptation of training and evaluating models at the same ...