Distributed AI Systems

9

Production LLM Serving Stack

Building end-to-end production systems for reliable LLM serving

 

Everything fails all the time.

Werner Vogels, CTO of Amazon

Anatomy of a production LLM serving system

In the previous chapters, we explored how to train large models at scale using distributed techniques like DDP, FSDP, DeepSpeed, and Megatron-LM. We also examined inference engines like vLLM and SGLang that optimize single-node and multi-node inference. Now it's time to put these pieces together into a complete production system.

AI model serving encompasses a broad spectrum of workloads. Traditional machine learning models—gradient boosted trees, linear models, and small neural networks—are typically served on CPUs with frameworks like TensorFlow ...