Evals for AI Engineers

Chapter 10. Interfaces for Human Review

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 10th chapter of the final book. If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

Automated evaluators (Chapter 5) handle most of the monitoring, but humans still review a sample of traces regularly to catch issues the evaluators miss and to keep evaluators calibrated as the team’s understanding of quality evolves (the criteria drift we discussed in Chapter 4). The question is how to do that review efficiently. LLM traces are complex. A single trace can include the user query along with multiple tool calls with structured arguments, retrieved documents, intermediate reasoning, and the final output. Reviewing these in a spreadsheet is painful, so we have found it very useful to build our own review interfaces. Not only is it easier to get through more traces, we end up actually wanting to look at more of our data, for as long as we are able to keep discovering new failure modes.

In this chapter, you will learn:

• Why custom review interfaces work better than spreadsheets and generic tools

• Key interface principles from Human-Computer Interaction (HCI) that apply to reviewing LLM agent outputs ...