AI Agents: The Definitive Guide

Chapter 9. Customized and Advanced Evaluation of Agentic Systems

In the previous chapter, you learned how to observe and diagnose agent behavior through stress testing, red-teaming, and production monitoring with traces. Tracing platforms are essential for observing agent behavior, but tracing alone is not the same as evaluation. A system can be fully instrumented and still leave you without a reliable way to judge whether a new model, prompt, or orchestration change actually improves the application. Unless you convert production evidence into structured evaluation cases, you will remain dependent on ad hoc inspection and isolated user feedback, which is usually sparse and incomplete.

This is why, in this chapter, you will build on the tracing foundation from the previous chapter and learn how to design your custom benchmark. Custom benchmarks are valuable because they allow you to test alternative models, fallback systems, prompt changes, or architectural modifications under controlled conditions before redeployment. This creates a continuous improvement loop where evaluation data from real usage directly informs future benchmarking and system updates. You will see this pattern across three settings: production traces turned into regression benchmarks, repositories turned into executable coding-agent benchmarks, and multimodal trajectories used to evaluate long-horizon agent behavior.

In addition to custom benchmarks, you will also see how to create evaluation data from your ...