Hands-On LLM Serving and Optimization

Chapter 8. LLM Serving Frameworks

Over the previous chapters, we’ve explored the fundamentals of LLM serving—system design, service implementation, and practical optimization techniques. This chapter shifts to the foundation layer—the serving frameworks that implement and execute model inference with different optimization techniques under real production constraints. We’ll discuss four widely adopted open source serving frameworks you’re likely to encounter in the wild: vLLM, TensorRT-LLM, SGLang, and llama.cpp. Each has a distinct philosophy, hardware footprint, and battle-tested technology, and is backed by active communities and growing production usage.

Because it’s the most broadly applied framework, we’ll take a deep dive into vLLM—its architecture, initialization and model-execution process, request and token-level scheduling, and layered optimization strategy. Understanding vLLM’s internals will give you strong intuition for how LLM frameworks work in practice and make it easier to evaluate the trade-offs in other frameworks.

Next, we’ll cover the remaining frameworks with concise, decision-oriented overviews and short examples. We’ll close the chapter with the evaluation method we use to compare serving frameworks.

After reading this chapter, you’ll have a solid grasp of what LLM serving frameworks are, why we need them, how they work under the hood, and how to evaluate them for your use case. In the next chapter, we’ll put the optimization techniques and serving framework ...