AI Evals in Practice

Chapter 9 — Human-in-the-Loop Annotation and Labeling Ops

Automated scorers — deterministic and LLM-based — are the backbone of a scalable eval system. But they have limits. Deterministic scorers cannot judge nuance. LLM judges have biases, cannot catch novel failure modes, and need calibration. At some point, you need a human to look at the output and say “this is good” or “this is wrong.”

Human evaluation is not a fallback — it is a critical component of a production eval system. It serves three purposes: calibrating your automated scorers, handling cases where automated scorers disagree, and evaluating subjective qualities (tone, helpfulness, brand alignment) that no automated system reliably measures.

The challenge is making human evaluation ...