Evals for AI Engineers

Chapter 2. LLMs and Evaluation Basics

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 2nd chapter of the final book. If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

In Chapter 1, we introduced evaluation and its role. Before we can evaluate, we need something to evaluate. This chapter walks through building a first application and understanding its moving parts.

In this chapter, you will learn:

• How to build a simple LLM agent from a single API call

• The components of LLM agents, from single calls to complex agent loops

• How to write initial prompts

• The two main evaluation modes we use throughout this book:

  • Absolute evaluation: judging whether one configuration (e.g., a specific prompt, model, or tool setup) is correct enough to ship.

  • Comparative evaluation: comparing multiple options (A/B tests, pairwise, ranking, leaderboards) to decide which is better.

By the end of this chapter, you will have a clear mental model of what makes up an LLM agent and some of the types of evaluation you might do for your application.

Components of LLM Agents

In Chapter 1, we defined an LLM agent broadly: it takes input, calls a language model one or more times, possibly invokes tools ...