Evals for AI Engineers

Chapter 7. Evaluating Retrieval-Augmented Generation

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 7th chapter of the final book. If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

The previous chapters covered evaluation techniques that apply broadly across LLM applications. Now we turn to a specific architecture that introduces its own evaluation challenges: RAG.

In this chapter, you will learn:

• How to build evaluation datasets for retrieval using synthetic query generation

• How to measure retrieval quality with Precision@k, Recall@k, MRR, and NDCG@k

• How to tune chunking strategies with grid search

• How to evaluate generation quality using faithfulness and relevance

RAG architectures are pervasive across customer support, enterprise search, scientific QA, and coding assistance. Evaluating RAG requires more than measuring final answer correctness. Each stage of the pipeline (query construction, retrieval, reranking, and generation) can independently introduce failure modes.¹ For example, failures across these stages are interdependent. If retrieval returns the wrong documents, the generator has no way to produce a correct answer. If the user’s query is ambiguous, retrieval ...