AI Engineering Interviews

Chapter 13. LLM Inference

You’ve spent months and millions training a 70B-parameter model. Now a user types “Hello” and your system needs to respond in under a second. That single reply kicks off inference: the process of generating output tokens from a prompt. At scale, inference is where the money goes; for a widely deployed model, the serving bill can exceed the original training cost within weeks.

In this chapter we explain how LLM inference works and how production systems make it fast, scalable, and cost-efficient. We start with the fundamentals of inference. We then cover optimization techniques to accelerate inference, and finally we conclude the chapter with deployment topics, including multi-GPU parallelism, continuous batching, model routing, and more.

Inference Fundamentals

We begin with the two-phase process by which LLMs generate text.

Prefill and Decode

When a user types “Summarize this article in three bullet points” and hits Enter, the model’s work begins in two distinct phases.

Prefill phase: The model processes all of the user’s input tokens in parallel, computing the key and value vectors that form the KV cache (explained in Section “Attention and Memory Optimization”) for every prompt position. The final prompt position also produces the logits from which the first output token is selected. From the user’s perspective, prefill is the pause between pressing Enter and seeing the first token appear.

Decode phase: Starting with the first output token produced ...