Chapter 4. Distributed Inference: Getting More Out of Your Infrastructure
The dashboard told the team the date. The date arrived early.
Chapter 3 ended with a single serving engine that held. The 300B-class assistant met its target time to first token. Streams were smooth. The GPUs earned their cost. The benchmark even drew a clean line around the single-node boundaries: the concurrency where the KV cache filled and preemptions began, the context length where prefill latency stopped being hideable, and the margin that adoption was visibly eating. The team could read the failure date directly from its own monitoring dashboard.
Then a new product group onboarded two weeks early. Its traffic was not huge, but it landed on top of the morning peak and brought larger knowledge bases, longer retrieved contexts, and a unique adapter. Performance did not degrade smoothly. Time to first token moved in steps, and streams that had been fluid started stalling for seconds. The on-call engineer recognized the Chapter 3 symptoms: cache saturation, sequence eviction, and ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access