Generative AI on Kubernetes

Part I. Inference

Part I discusses the key aspects to consider during the deployment and execution of a generative AI model. This is addressed first because, unlike predictive AI models, you typically don’t start from scratch by creating an entirely new foundation model.

The creation of a foundation model is an extremely resource, time, and data intensive activity that adopts similar techniques but at a larger scale. A very limited number of companies perform similar activities, so it is not covered in this book.

This phase of the model lifecycle is not new to the AI space; model serving has always been a core aspect since the beginning of AI adoption in production contexts. However, the size and complexity of these new types of models introduce a significant set of new challenges. Beyond its historical importance, inference has become the primary entry point for modern AI adoption. AI engineers typically begin by integrating pre-trained models into their existing applications through inference APIs, allowing them to infuse traditional software with AI capabilities without requiring deep machine learning expertise. Moreover, inference infrastructure serves as the foundational layer for agentic AI systems, where multiple models and tools must work together seamlessly to accomplish complex tasks.

Generative AI is a very active field, and this book doesn’t aim to be a comprehensive list of available projects, runtime, or tools. The development is far from being done, with ongoing ...