Hands-On LLM Serving and Optimization

Foreword

The rise of large language models (LLMs) marks a fundamental shift in how we build and interact with software systems. What began as a series of research breakthroughs has rapidly become the backbone of modern AI applications—from copilots to autonomous agents. Model development and deployment have advanced at an extraordinary pace. Yet today, the industry is converging on a new reality: model inference at scale has become the defining challenge.

How efficiently we serve models—how quickly, how reliably, and at what cost—now determines whether AI can deliver real value to customers. This is no longer a secondary concern. It is the primary focus for companies building AI-driven products and businesses.

Hands-On LLM Serving and Optimization addresses this challenge with clarity and practical insight.

Chi Wang and Peiheng Hu bring a rare combination of deep technical expertise and real-world experience. They have worked together for over eight years at the forefront of building and operating large-scale AI systems. Their perspective is grounded in production environments, where trade-offs between latency, throughput, cost, and reliability must be carefully balanced.

One of the strengths of this book is its structured approach. It begins with the fundamentals of model serving—covering core system-design principles that apply broadly across machine learning systems—before moving into the unique challenges of LLMs. This progression makes the book accessible to readers who are ...