AI Evals in Practice

Chapter 5 — LLM-as-Judge: Design, Calibration, and Bias

Deterministic scorers catch format errors, missing fields, and structural problems. They cannot tell you whether an answer is helpful, whether a summary captures the key points, or whether a chatbot response has the right tone. For those judgments, you need a judge that understands language. That judge is another LLM.

LLM-as-judge is the technique of using a language model to evaluate the output of another language model. The judge receives the input, the output, and a rubric describing what “good” looks like, and it produces a structured assessment. When done well, LLM-as-judge reaches 80-90% agreement with human evaluators — comparable to inter-annotator agreement between humans themselves. ...