Scaling AI Inference: From Tokens to Throughput

What you’ll learn and how you can apply it

• Distinguish model serving from full AI-powered product serving and explain why one user task may create many model operations
• Estimate token volume, cost per task, and cost per successful task for common GenAI workloads
• Set workload-aware latency and throughput targets using TTFT, TPOT/time-between-token, tail latency, concurrency, and goodput
• Explain prefill, decode, KV cache, batching, prefix caching, and memory headroom in a production serving engine
• Identify when optimization choices such as quantization, speculative decoding, routing, caching, or scale-out may help or hurt
• Create a lightweight benchmark and capacity-planning checklist for vLLM-style and Kubernetes-native inference deployments
• Treat RAG, agentic workflows, and eval loops as inference workload multipliers that affect latency, cost, and reliability
• Decide the next practical step for an inference workload: simplify, cache, batch, route, quantize, scale out, or benchmark more deeply

Course description

AI inference is where generative AI systems become operationally real: Every successful task creates token volume, latency expectations, memory pressure, routing choices, and recurring cost. This three-hour live course with Bruce Hopkins gives AI/ML engineers, platform engineers, and SREs a practical framework for reasoning about inference at scale before they commit to a model, hardware profile, or serving architecture.

You’ll follow a recurring enterprise support and code-assistant scenario. You’ll translate token usage into successful-task economics; connect TTFT, time-between-token, throughput, concurrency, and goodput to user experience; and evaluate serving-engine choices such as KV cache management, continuous batching, prefix caching, quantization, distributed inference, routing, and observability.