An Illustrated Guide to AI Agents

Chapter 7. Evaluating Agents

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 7th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

The previous chapters built up the components of an agent one by one: how it reasons, how it uses tools, how it stores and retrieves memory, and how it plans across multiple steps. By now you have a mental model of how these pieces fit together to form the system we call an AI agent. The natural next question to ask is, how do we know if any of it is working?

That’s what this chapter is about, as we can see in Figure 7-1. Evaluation is how you move from “it seems to work” to “I have evidence it works.” It’s also how you catch the moment it stops working. Evaluations are one of the most underinvested areas in agent development. Teams often rely on manual testing and intuition longer than they should, then find themselves unable to ship improvements confidently or diagnose regressions when they appear.

Figure 7-1. This chapter covers how LLMs and agents are evaluated ...