Hands-On LLM Serving and Optimization

Chapter 2. Large Language Model Serving

In the previous chapter, we introduced the concept of models, model serving, and common serving paradigms. In this chapter, we shift our focus to the specific challenges and techniques involved in serving LLMs.

One of the most common barriers for those entering the world of model serving is the sheer complexity of modern serving systems. With rapidly evolving model architectures, training algorithms, and tooling—combined with layers of production infrastructure like monitoring, scaling, security, CI/CD pipelines, and service dependencies—the experience can quickly become overwhelming. The result is that many engineers and researchers lose focus, getting bogged down in system details before they can grasp the core principles.

To address this, our approach is to start from the fundamentals. We begin with the bare minimum code required to serve an LLM and build up from there. This helps establish a strong mental model of how token generation works and why serving LLMs poses unique challenges. From this foundation, we’ll gradually expand into more advanced topics like system architecture, performance optimization methods, and infrastructure choices in the remaining chapters. In this chapter, we start with:

• The basic architecture of LLMs, including the token generation process and attention mechanisms

• What happens under the hood during inference through hands-on code examples

• The core concepts behind LLM serving, such as prefill, decode, ...