Hands-On LLM Serving and Optimization

Chapter 1. Introduction to Model Serving and Optimization

Over the past decade, AI systems have evolved from offline research prototypes into real-time, user-facing capabilities embedded in everyday products. Modern AI workflows span the full lifecycle—from data collection and model training to deployment, monitoring, and continuous iteration—and this lifecycle has accelerated dramatically with the rise of deep learning and large language models (LLMs). While training increasingly powerful models has captured much of the attention, delivering those models reliably and efficiently in production has become just as critical.

At its core, model serving is a process that addresses the challenge of making AI models accessible to end users, applications, and systems, working through APIs, web services, or integrated workflows to generate predictions (called inferences) on new, unseen data.

To draw a simple analogy, to businesses of all kinds—whether they aim to deliver AI capabilities to their customers or enhance operational efficiency—model serving is a form of supply chain. A trained model has little business value unless it can be delivered to users with the right latency, reliability, and cost characteristics. For example, Amazon and Netflix use model serving to update customer recommendations instantly as users browse. Banks use model serving to block fraudulent transactions during online shopping checkouts, and airline chatbots use it to provide instant flight updates and rebooking ...