AI Evals in Practice

Chapter 7 — Evaluating RAG: Retrieval and Generation Metrics

Retrieval-Augmented Generation is one of the most deployed LLM patterns in production. The idea is simple: retrieve relevant documents, pass them to the model as context, and generate an answer grounded in those documents. The implementation is deceptively complex — and the evaluation is where most teams discover just how complex.

A RAG system can fail in at least four distinct ways: the retriever can return irrelevant documents, the retriever can miss relevant documents, the generator can ignore the retrieved context and hallucinate, or the generator can faithfully use the wrong context. Each failure mode requires a different metric, and existing observability tools measure none of ...