Hands-On Generative AI with Transformers and Diffusion Models

Appendix B. LLM Memory Requirements

Models come in all sizes! Llama 3.1, for example, was released with 8B, 70B, and 405B variants. To load and use an LLM, you need enough memory to store the model. The number of parameters and their precision, among other factors, influence the memory requirements for an LLM.

What can you do if you do not have enough memory? Try these options:

• Reduce the precision of the model you are using. Rather than using float16, you can use int8.

• Use a smaller model. There are many high-quality small models.

• Unload parts of the model that you are not using. This can be done with CPU RAM offloading, a common technique to reduce a model’s memory requirements at the cost of slower inference speeds. What happens if there is not enough memory? We can then store the remaining model parts on the disk and load them as needed. Fortunately for us, the accelerate library takes care of this via device_map="auto", which will automatically offload parts of the model as needed.

Inference Memory Requirements

You can roughly estimate the memory requirements as follows:

$$\text{GPU}\text{memory}\text{needed}=\text{Number}\text{of}\text{parameters}\times \text{Bytes}\text{per}\text{parameter}$$

The bytes per parameter depends on the precision used. Without going into too much detail, Table B-1 shows the memory needed to load 2B, 8B, 70B, and 405B models using different levels of precision (float32, float16, int8, int4, and int2).

Table B-1. Inference memory requirements for models and levels of precision

Model ...