Evals for AI Engineers

Chapter 5. Implementing Automated Evaluators

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 5th chapter of the final book. If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

In Chapter 3, we identified failure modes by reading traces. In Chapter 4, we aligned the team on what counts as a failure. Now we automate that measurement so we can track failure rates at scale without manually reviewing every trace.

In this chapter, you will learn:

• How to translate failure modes into precise, automatable metrics

• When to use code-based checks versus LLM-as-Judge evaluators

• How to build and validate an LLM-as-Judge using training, dev, and test sets

• How to estimate the true success rate of your agent in production

• How to quantify uncertainty using bootstrapping and confidence intervals

Automated evaluators can compute various types of metrics.¹ Some metrics are reference-free (as defined in Chapter 2), assessing inherent qualities of an output or its adherence to certain rules without needing a “golden” or ground-truth answer. Others are reference-based, comparing the agent’s output to a known correct or ideal response. For many failure modes, we can conceptualize and implement ...