Evals for AI Engineers

Chapter 2. LLMs and Evaluation Basics

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 2nd chapter of the final book. If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

In Chapter 1, we motivated why evaluation is essential. But before we can evaluate, we need something to evaluate! This chapter equips you with everything you need to get a first application running and to understand its moving parts.

Specifically, we will:

• Ground the discussion with a simple example application.

• Break down the components of an LLM application—​from the simplest single call to complex agent loops.

• Describe some fundamentals of creating prompts or initial versions of these components.

• Clarify the two main evaluation modes we will use throughout this book:

  • Absolute evaluation: judging whether one configuration (e.g., a specific prompt, model, or tool setup) is correct enough to ship.

  • Comparative evaluation: comparing multiple options (A/B tests, pairwise, ranking, leaderboards) to decide which is better.

By the end of this chapter, you will have a clear mental model of what makes up an LLM application and some of the types of evaluation you might do for your application.

Components of ...