book

Evals for AI Engineers

by Shreya Shankar, Hamel Husain

October 2026

Intermediate to advanced

240 pages

6h 21m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Preface
What This Book CoversWho Should Read This BookA Note on the Pace of ChangeWhat This Book Doesn’t CoverHow This Book Is OrganizedConventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
I. Foundations
1. Introduction
What Is Evaluation?Evaluation Throughout the Lifecycle of an LLMPretrainingPost-trainingApplicationEvaluation in ApplicationsThe Three Gulfs of LLM Application DevelopmentThe Gulf of Comprehension (Between the Developer and Data)The Gulf of Specification (Between the Developer and the LLM Agent)The Gulf of Generalization (Between the Data and the LLM Agent)Why LLM Agent Evaluation Is ChallengingThe LLM Evaluation Lifecycle: Bridging the Gulfs with EvaluationAnalyzeMeasureImprovePutting It Together
2. LLMs and Evaluation Basics
Components of LLM AgentsSingle LLM CallConversationsRetrievalToolsAgent LoopsFundamentals of PromptingPrompts for Single LLM CallsRetrieval FundamentalsTools and AgentsEvaluation SetupsAbsolute EvaluationComparative EvaluationPutting It Together
II. The Analyze Phase
3. Error Analysis
Establishing TerminologyCreating a Starting Dataset of TracesDefining DimensionsGenerating Tuples and QueriesOpen Coding: Reading and Labeling TracesAxial Coding: Structuring Failure ModesLabeling Traces with Structured Failure ModesIteration and RefinementCommon PitfallsPutting It Together
4. Collaborative Evaluation Practices
When a Single Expert Is EnoughA Collaborative Annotation WorkflowMeasuring Inter-Annotator AgreementPercent Agreement: Simple but MisleadingCohen’s Kappa: Adjusting for ChanceInterpreting Kappa ScoresPython ImplementationWhen to Use Other MetricsFacilitating Alignment SessionsPreparing for the SessionRunning the SessionTechniques for Improving the RubricWhat to AvoidEscalationSave EverythingCommon PitfallsSkipping Independent AnnotationStarting with a Vague RubricUsing Only Percent AgreementIgnoring Temporal DriftSynthetic-Annotator CollusionFocusing on Past Labels Instead of Future ConsistencyPutting It Together
III. The Measure Phase
5. Implementing Automated Evaluators
Defining the Right Metrics (What to Measure)Implementing Metrics (How to Measure)Writing LLM-as-Judge PromptsData Splits for Designing and Validating LLM-as-JudgeIterative Prompt Refinement for the LLM-as-JudgeWhen to Stop RefiningIf Alignment StallsEstimating True Success Rates with Imperfect JudgesStep 1: Measure Judge AccuracyStep 2: Observe Raw Success RateStep 3: Correct the Observed Success RateStep 4: Quantify Uncertainty with a BootstrapPython Code for Estimating Success RatesOptional: Group-wise Metrics for Evaluating Multiple OutputsCommon PitfallsPutting It Together
6. Evaluating Multi-Turn Conversations
Evaluating at Different LevelsSession LevelTurn LevelConversational Coherence and MemoryPractical Strategies for Multi-Turn EvaluationCollecting TracesIsolating FailuresOperationalizing Session-Level SuccessUser Simulators for Generating Test ConversationsStratifying by Conversation LengthPerturbation TestingAutomated Evaluation of Multi-Turn TracesCommon PitfallsPutting It Together

7. Evaluating Retrieval-Augmented Generation
OverviewSynthetically Generating Query-Answer PairsCreating Harder Synthetic QueriesFiltering Synthetic QuestionsMetrics for Retrieval QualityPrecision@k and Recall@kMean Reciprocal RankNormalized Discounted Cumulative Gain (NDCG@k)Evaluating and Optimizing Chunking StrategiesTuning Chunk Size and OverlapOther Chunking ApproachesEvaluating Generation QualityCommon PitfallsPutting It Together
8. Evaluating Tool Use and Complex Agents
What Are Tools and Why Do LLMs Use Them?The Spectrum of AgencyEvaluating Tool CallsStage 1: Tool SelectionStage 2: Argument GenerationStage 3: Tool ExecutionStage 4: Output HandlingWhy Evaluate Each Stage SeparatelyDebugging Multi-Step TracesTraceabilityFrom Traces to Systemic PatternsThe Transition Failure HeatmapInvestigating HotspotsEvolving the Heatmap over TimeHandling Complex Input DataImagesLong DocumentsPDF DocumentsAdversarial and Security EvaluationCommon PitfallsPutting It Together
IV. Production and Improvement
9. Continuous Integration and Deployment for LLM Agents
CI: Building a Safety Net Against RegressionsCommon Mistakes with CIRevisiting Prompts When Models ImproveCD and Online MonitoringObservabilityRunning Automated Evaluators in ProductionGuardrailsJudge DriftThe Continuous Improvement FlywheelCommon PitfallsPutting It Together
10. Interfaces for Human Review
The Case for Custom Review InterfacesEssential Interface ElementsFeatures That Improve Review Speed and QualityCase Study: EvalGenSelecting Traces for Human ReviewRandom SamplingUncertainty SamplingFailure-Driven SamplingNavigating Trace Groups and Discovering PatternsClusteringSearch and Similarity ToolsIntegrating Human Review into Engineering WorkflowsExample Walkthrough: Reviewing Real Estate Assistant EmailsStarting with a SpreadsheetA Simple Custom UIAn Advanced Review InterfaceCase Study: DocWrangler for Prompt RefinementCommon PitfallsPutting It Together
11. Data Analysis for Traces
EDA and Clustering to Surface HypothesesClusteringBeyond ClusteringFrom Clusters to Confirmed HypothesesSemantic Search to Find Related TracesUsing AI Coding AssistantsCommon PitfallsPutting It Together
12. Improving LLM Agents
Accuracy OptimizationLow Effort: Prompt RefinementMedium Effort: Changing How the Agent WorksHigh Effort: Model-Level ChangesCost OptimizationMatch Model Size to Task ComplexityCut Token UsageMake Prompts Cache-FriendlyBatchingDistillationModel CascadesRoutingClosing: The Lifecycle ContinuesChoosing an Improvement StrategyNext Steps
Bibliography
Index
About the Authors

Overview

Stop using guesswork to find out how your AI applications are performing. Evals for AI Engineers gives you a practical framework for systematically testing, measuring, and improving AI applications, especially those built with LLMs. Written by Shreya Shankar and Hamel Husain, AI engineers with extensive experience across 35+ AI products and cutting-edge research, this book focuses on the techniques teams actually need to build reliable AI systems in production.

Ideal for software engineers, technical product managers, and technical leads, this hands-on guide shows you how to build an evaluation process that improves alongside your product.

Find and prioritize failures through systematic error analysis and representative test sets
Build trustworthy evaluations using code based checks, LLM judges, human feedback, and synthetic data
Improve production systems over time by monitoring regressions and optimizing quality, latency, and cost

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9798341660717Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills