Evals for AI Engineers

Chapter 1. Introduction

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 1st chapter of the final book. If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

The past several years have seen rapid advances in the development and deployment of large language models (LLMs). Organizations are embedding LLMs into software products for customer service, content creation, decision support, and information extraction, among many others ([Link to Come], Bommasani et al. 2021, [Link to Come], Zaharia et al. 2024). We call these software products LLM applications, and the LLM-driven component inside each one (the part that makes decisions, calls tools, and produces output) the LLM agent.

What Is an “Agent”?

We use “agent” broadly throughout the book. It covers the full range from a single LLM call with no tools (the simplest case) to a system that runs in a loop, calls many tools, and makes many decisions per request. Chapter 2 walks through this range with concrete examples.

The distinction between agent and application is useful for evaluation. Some failures come from the agent itself (wrong tool call, incorrect reasoning, bad output format). Others come from the application around it (broken ...