Evals for AI Engineers

Chapter 3. Error Analysis

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 3rd chapter of the final book. If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

Figure 3-1. Anthropic’s visualization of the iterative process for creating strong empirical evaluations. This cycle emphasizes developing test cases, engineering and refining prompts, and testing against both initial cases and held-out evaluations. Image source: Anthropic Anthropic 2024.

The process of developing robust evaluations for LLM applications is inherently iterative. It involves creating test cases, assessing performance, and refining the system based on those observations. High-level guides, such as Anthropic’s documentation on creating empirical evaluations for Claude Anthropic 2024, often depict the evaluation process as a cycle of developing test cases, engineering prompts, testing, and refining (Figure 3-1).¹ This chapter covers the Analyze step of our Analyze-Measure-Improve lifecycle (Figure 1-2). We walk through how to bootstrap an initial dataset, read traces, and build a structured understanding of where ...