Sustainable AI

Chapter 5. Automating Model Selection

The adage “pull out all the stops,” meaning you exhaust all resources to achieve a goal, might apply to how models are currently designed in DL. This work involves adjusting multiple configurations of a DL model, somewhat like an organist using a variety of stops to produce different sounds. Obtaining the appropriate class, configuration, and parameters for a particular downstream task from the massive space of possibilities is known as model selection. This step of model selection is tedious, and requires several orders of magnitude more compute resources than training the final model, as illustrated in Figure 5-1. Further, the process of model selection in DL is known to be as much art as it is science, requiring significant human effort.¹

Figure 5-1. The proverbial “tip of the iceberg” captures the seldom-addressed costs of performing model selection in DL. The vast hypothesis space increases the model selection costs many folds over compared to the training cost of a single model.

In this chapter, we will understand why model selection is a computationally expensive process, look at some fundamental concepts of model selection, identify the AI waste involved in this step, and try out various tools for model selection. We will look at methods for automated model selection, with the objective of improving the overall efficiency of AI methods ...