Scalable Kubernetes Infrastructure for AI Platforms

Chapter 4. Model Deployment and Monitoring

In the previous chapters, you learned about model customization techniques, including fine-tuning and training, and about making training and evaluation repeatable. Once you’ve achieved the results you are looking for with your model, it’s time to deploy your model to production.

This chapter will prepare you for model deployment and serving by giving you an overview of the major technologies and techniques used with Kubernetes to deploy and monitor machine learning models. While we will focus on specific techniques relevant to large language models (LLMs) and generative AI, much of this chapter will also apply to traditional machine learning models.

Overview of LLM Serving

Model serving is the act of processing inference requests in real time, which requires deploying an already trained model to some location suitable for receiving these requests. At a high level, model serving involves packaging the model, deploying it on hardware accelerators like GPUs or CPUS, exposing APIs for users to query the model, and enabling metrics for monitoring and alerting. The components of a model-serving system include model-serving platforms, model-serving runtimes, and metric gathering and monitoring systems. Typically, an API gateway and load balancer to handle bursts of traffic for model queries is also included.

The model-serving platform component retrieves the model from storage (such as Amazon S3 or a local persistent volume) and then performs ...