Generative AI on Kubernetes

Chapter 2. Model Data

One of the most fundamental challenges when running LLMs on Kubernetes is managing the sheer size of the model data. LLMs can range from a few gigabytes to nearly a terabyte in size, and efficiently bringing this data into a cluster where runtimes can access it requires careful consideration.

The main portion of those models consists of the model parameters and can be extremely large. Table 2-1 lists the number of parameters and size of some more prominent available models that you can run yourself. There are many more, but from this selection you can already see a wide range of variations. These range from large models that are likely impractical for on-demand use, to more lightweight models that can be run on your own cluster and easily downloaded when needed.

Table 2-1. Open source models and their sizes

Name	Vendor	Parameters	Size
Llama 4 Maverick	Meta	400 billion (MoE, 17B active)	~800 GB
DeepSeek-V3	DeepSeek	671 billion (MoE, 37B active)	~700 GB
Llama 3.1 405B	Meta	405 billion	~750 GB
Qwen3-235B	Alibaba	235 billion (MoE, 22B active)	~118 GB
Mixtral 8x22B	Mistral	141 billion (MoE, 39B active)	~88 GB
GPT-OSS 120B	OpenAI	117 billion (MoE, 5B active)	~70 GB
Gemma 2 27B	Google	27 billion	~54 GB
Granite 13B	IBM	13 billion	~26 GB
Falcon 2 11B	TII	11 billion	~22 GB
Mistral 7B	Mistral	7 billion	~14 GB

Even smaller models can pose significant challenges for Kubernetes administrators when managing them efficiently within a cluster. ...