Evals for AI Engineers

Chapter 9. Continuous Integration and Deployment for LLM Agents

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 9th chapter of the final book. If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

In July 2023, researchers at Stanford and Berkeley published a study tracking GPT-4’s behavior over time (Chen et al. 2024). They ran the same math tasks on the GPT-4 API in March 2023 and again in June 2023, and found that accuracy on identifying prime numbers dropped from 84% to 51%. OpenAI had updated the model behind the gpt-4 API endpoint without any announcement. Teams that had been calling gpt-4 in production woke up one morning with a model that behaved differently on their tasks, and they had no way of knowing until users complained.

This is the kind of problem that continuous evaluation is designed to catch. Getting an LLM agent to work well once is hard enough. Keeping it working over time is a different challenge entirely. User needs change, data distributions shift, and model providers push updates without warning. A nightly regression suite that re-evaluated our prompts against a fixed set of test cases would have caught that accuracy drop before it reached production.

In this ...