Generative AI on Kubernetes

Chapter 1. Deploying Models

Running models within your own cluster becomes necessary when real data cannot leave it due to privacy laws or compliance requirements, or when you need greater control over model deployment and performance.

There are many different models on the market; many of them are open source and freely available for commercial use. Hugging Face is the largest community where you can find not only models but also datasets and libraries. For a list of current open source large language models, see Chapter 2.

Regardless of where you obtained the model, whether it’s open source or not, there are aspects of deploying the model on Kubernetes that aren’t specific to the model itself. However, some aspects require careful analysis of the model to determine the best approach.

This chapter describes different approaches and patterns for managing the lifecycle of your model at runtime, with a focus on some of the most-used runtimes for LLMs. Before diving into deployment details, see the following sidebar for background on the Transformer architecture that powers most modern LLMs.

Transformer Architecture and Attention Mechanism

Generative AI is a vast field, and the maturity levels of different model classes vary significantly, with text generation models being the most widely used and optimized.

Modern LLMs are predominantly based on Transformer architecture or a derivative (like the mixture of expert approach), although Transformers are also used beyond text for vision ...