Evals for AI Engineers

Chapter 8. Evaluating Tool Use and Complex Agents

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 8th chapter of the final book. If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

So far, we have not thought too carefully about how much autonomy our agents have, or how many tools they use. The techniques in previous chapters (error analysis, automated evaluators, CI/CD) apply whether the agent makes one LLM call or twenty. But as agents become more complex (calling external functions, running multi-step loops, making decisions about what to do next), evaluation gets harder. Errors cascade across steps, and when something goes wrong in the final output, it can be difficult to tell which step caused the problem.

In this chapter, we focus on how to evaluate these systems systematically. We start by defining what tools are and how LLMs use them. We then discuss the spectrum of agency, which determines how much autonomy the system should have. From there, we walk through a four-stage framework for evaluating individual tool calls, a method for debugging multi-step traces, and considerations for handling complex input data like images, long documents, and PDFs.

What Are Tools and Why ...