Generative AI on Kubernetes

Chapter 5. Model Observability

In Chapter 1 we learned how to deploy an LLM in Kubernetes, starting from scratch with a simple coding example. The full stack included a model server, vLLM, to optimize the execution of the model, and a Model Server Controller, KServe, to manage the integration with Kubernetes and the lifecycle of the deployment.

Then in Chapter 2 we focused on LLM model data, with the complexity and options that are available today to manage the size of similar models. We are getting closer and closer to a full production setup where the LLM workload is fully managed and automated so that it can be executed side by side with the other workloads (i.e., traditional applications), all managed by Kubernetes.

Kubernetes orchestrates container execution through a declarative API, using controllers and reconciliation loops to self-heal workloads in an eventually consistent way. Everyone with Kubernetes experience knows that this approach doesn’t replace proper observability and monitoring. These capabilities allow you to quickly react when something cannot be solved automatically. This principle applies to LLMs too. It is critical to monitor a model server but, given the nature of LLMs, it is not equivalent to monitoring traditional applications.

LLMs differ significantly in how they produce workload as compared to traditional microservices with few endpoints, where the workload is mainly driven by number of requests and speed of query on data. LLMs are different even ...