Chapter 3. Enterprise Serving Engines
Every serving platform in this book, however far the later chapters stretch it, starts as one box: a single model on a single node behind a single endpoint. This chapter is about understanding that box before we build anything around it.
The team behind the unified assistant arrives here with its homework done. Chapter 2 left it with classified workloads: support questions that are sensitive to latency, lean on retrieval, and occasionally touch restricted content; code questions that arrive in bursts, carry long pasted contexts, and reward fast streaming because a developer is waiting inside an editor; and batch summarization jobs that nobody watches in real time. Now someone has to deploy a model.
Chapter 3 keeps the same 300B candidate as the working example. Treat it as an intentional stress case, not a normal single-node recommendation: the point is to make serving-engine pressure, KV-cache growth, batching tradeoffs, and benchmark questions visible before any team turns the board mandate into a sizing plan.
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access