Evals for AI Engineers

Chapter 1. Introduction

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 1st chapter of the final book. If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

The past several years have seen rapid advances in the development and deployment of large language models (LLMs). Organizations are embedding LLMs into software products for customer service, content creation, decision support, and information extraction, among many others (Bommasani et al. 2021, Zaharia et al. 2024). We call these software products LLM applications, and the LLM-driven component inside each one (the part that makes decisions, calls tools, and produces output) the LLM agent.

What Is an Agent?

We use “agent” broadly throughout the book. It covers the full range from a single LLM call with no tools (the simplest case) to a system that runs in a loop, calls many tools, and makes many decisions per request. Chapter 2 walks through this range with concrete examples.

The distinction between agent and application is useful for evaluation. The agent is the LLM-driven component that reasons, calls tools, and produces output. The application is everything around it: the user interface, the database, the API layer, the deployment ...