Evals for AI Engineers

Chapter 1. Introduction

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 1st chapter of the final book. If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

The past two years have seen rapid advances in the development and deployment of large language models (LLMs). Organizations are now embedding LLMs in critical pipelines in applications: customer service, content creation, decision support, and information extraction, among others Bommasani 2021, Zaharia 2024. However, adoption is outpacing our ability to systematically evaluate LLM applications Ward 2024.

Unlike traditional software, LLM applications do not produce deterministic responses, nor do they have formal specifications for the logic they should and should not perform. Their outputs aren’t predictable. Sometimes the outputs are obviously incorrect. Other times, it is hard to describe what is wrong. An output can be factually right but seem inappropriate (the “vibes are off”), or sound completely convincing while being totally incorrect. These ambiguities make evaluation fundamentally different from conventional software testing and even traditional machine learning (ML).¹

The core challenge of evaluation is as follows: ...