Sustainable AI

Chapter 6. Training Efficiency

In the Indian state of Karnataka, the 12th century Chennakeshava Temple complex features sculptures of deities and epic scenes carved exquisitely in stone (Figure 6-1).¹ The monumental effort required to complete these carvings is similar in some ways to the effort that goes into training modern DL models. Instead of meticulously chiseling away stones, iterative optimization algorithms such as stochastic gradient descent chisel away the trainable parameters of a deep neural network in an iterative manner to create impressive AI models.²

Figure 6-1. The stone-carved facade of the temple complex in Belur, India.

In Chapter 5, we explored methods for choosing DL model architecture. Given a specific model architecture, the training process ingests large amounts of training data to obtain models that can be useful for downstream tasks. This training process can be computationally intensive, as models with hundreds of billions of parameters have to work through datasets with as many as a trillion data points (as seen in Chapter 4). Due to these factors, the training compute required is following a mind-boggling trend. According to recent estimates, the compute FLOP required to train DL models has grown four to five times yearly from 2010 to May 2024, as shown in Figure 6-2, for some of the most popular AI models.

In this chapter, we will focus on how ...