Distributed AI Systems

6

Distributed Inference and vLLM

Serving large language models at scale with high throughput and low latency

 

Inference is the new web app.

Clayton Coleman, Distinguished Engineer at Google

From training to inference

The previous chapters covered how to train large models across multiple GPUs—state sharding with ZeRO and FSDP, computation sharding with Megatron's tensor and pipeline parallelism. But training is only half the equation. Once you have a trained model, you need to serve it to users, and serving brings an entirely different set of challenges.

Training optimizes for throughput: process as many tokens as possible per second, amortized over long training runs. Inference optimizes for latency and throughput simultaneously: users expect ...