book

Hands-On RAG for Production

by Ofer Mendelevitch, Forrest Sheng Bao

May 2026

Intermediate to advanced

358 pages

10h 52m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Includes

Quizzes

Foreword by Sharon Zhou
Foreword by Jim Dowling
Preface
What This Book Is AboutWho This Book Is ForWho This Book Is Not ForUsing Code ExamplesPrerequisitesNavigating This BookConventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
1. Introduction to Retrieval-Augmented Generation (RAG)
How Does RAG Work?The Blueprint of a RAG StackThe Ingestion FlowThe Query FlowExample: RAG with LangChainRAG Versus Other ApproachesRAG Versus “Chat with PDF”RAG Versus Fine-TuningKey Benefits of RAGRAG Is Scalable and EfficientRAG Helps Reduce HallucinationsRAG Enables ExplainabilityNear-Instant Addition and Removal of KnowledgeAccess Controls and SecurityRAG Use CasesVirtual Assistants and AI ChatbotsEnterprise Knowledge Management and Internal SearchAutomated Content Creation and Document SummarizationGenerating Attractive and Effective Personalized AdsQuestion-Answering SystemsMedical and Healthcare ApplicationsLegal and Compliance ResearchAdvanced RAGAgentic RAGMultimodal RAGRAG with Knowledge GraphsConclusion
2. The Base RAG Stack
RAG Stack FlowsThe Ingestion FlowThe Query FlowDocument ParsingExtracting Text from Various File Formats Document Parsing with Vision–Language ModelsCode Example: Parsing FilesText ChunkingChunking StrategiesCode Example: Chunking in PythonEmbedding ModelsWhat Is (an) Embedding?Selection Criteria for Embedding ModelsPractical Tips and ConsiderationsCode Example: Generating Embeddings with Sentence TransformersVector Databases and Vector SearchUnderstanding Vector-Based Similarity SearchApproximate Nearest Neighbor AlgorithmsVector DatabasesParameters to Consider When Using Vector SearchCode Example: Storing and Retrieving Vectors Using pgvectorGenerative LLMsLLMsRAG Prompt EngineeringEvaluating LLMs and Prompt TemplatesCode Example: Using Anthropic Claude to Generate Responses in RAGConclusion
3. Scaling Your RAG Stack
RAG at ScaleVolume and Complexity of DocumentsIndex FreshnessCost Management and OptimizationAdvanced Data IngestionHandling a Large Volume of DocumentsDealing with Inconsistent Data QualityHandling Large DocumentsExample: Splitting a Large PDF FileManaging Document Updates and RefreshAdvanced RetrievalThe Two-Stage Retrieval PipelineHybrid SearchRerankingImplementing GuardrailsGuardrails for AI SafetyPreventing Prompt Injection AttacksControlling Hallucinations in RAGDefining Hallucinations in RAGLLM Hallucinations Versus RAG HallucinationsHallucination DetectionHallucination CorrectionBuilding a Great RAG User ExperienceConsiderations for RAG User ExperienceMultimodal User InterfacesTools and Reference ImplementationsConclusion
4. Deploying RAG to Production
Challenges with RAG in ProductionResponse Quality and Reduced HallucinationsHigh LatencyData Security and PrivacyVendor Chaos and Integration WoesTeam and ExpertiseTotal Cost of OwnershipRAG EvaluationA Reference Production ArchitectureSuccessful Transition from Proof of Concept to ProductionSummarize What You Learned in the Proof of ConceptDefine Goals and RequirementsEnsuring Continued RAG SuccessConclusion
5. The RAG Platform
DIY Versus Platform RAGCore RAG CapabilitiesData SourcesRAG Sprawl and Centralized GovernanceCost and UpkeepDeployment OptionsExample RAG Platform: VectaraGetting StartedIngesting Data into VectaraRunning QueriesHallucination CorrectionOther RAG Admin API EndpointsConclusion
6. Evaluating Your RAG Application
How Does RAG Fail?Retrieval FailuresGeneration FailuresFailures Due to Inadequate Data IngestionSummary of RAG FailuresUsing LLMs for Evaluation: LLM-as-a-JudgeWhat Is LLM-as-a-Judge?How LLM-as-a-Judge WorksRAG Evaluation MetricsRetrieval MetricsGeneration MetricsBias and SafetyRAG Evaluation OfferingsOpen RAG EvalRetrieval-Augmented Generation AssessmentDeepEvalAmazon BedrockHuman FeedbackIntegrating RAG Evaluation in ProductionUsing LLM-as-a-Judge in ProductionOffline RAG EvaluationOnline RAG EvaluationSystem Metrics: Latency and UptimeLatency and ThroughputReliability and UptimeCost and Resource EfficiencyConclusion
7. From RAG to AI Agents
What Is an AI Agent?The Agentic StackSingle-Agent Versus Multi-Agent SystemsAgentic Use CasesAgents in Customer ServiceAgents in Financial ServicesAgentic AI in HealthcareAI Coding AgentsThe Agentic LoopTool CallingModel Context ProtocolModel Context Protocol ArchitectureMCP in Enterprise Agentic AIAgent-to-Agent CommunicationHands-On with Agentic AI FrameworksAI Chatbots Using LangChainDocument Generation Agent with LlamaIndexBuilding an Agent with VectaraBuilding a Multi-Agent System with CrewAIAgentic MemoryShort-Term Versus Long-Term MemoryImplementation Memory with Agentic RAGEnterprise Guardrails: Privacy and IntegrityEvaluation and Observability with AI AgentsAgentic ObservabilityTracing an AgentAgentic Observability MetricsTools for Agentic ObservabilityConclusion

8. Multimodal RAG
Documents with Embedded TablesWhy Are Embedded Tables Important?Extracting Tables from DocumentsWhy Naive Chunking Fails for TablesProcessing Tables for RAGDealing with Multi-Page TablesDocuments with Embedded ImagesThe Image Summarization ApproachMultimodal Retrieval with a Shared Embedding SpaceAudio and Video in RAGThe Baseline: High-Fidelity TranscriptionVisual Semantics: The “Red Button” ProblemProduction ConsiderationsComputational Economics and LatencyModality AlignmentThe Interface Layer: Visual CitationsSecurity, Privacy, and GovernanceDeep Observability, Tracing, and Security at ScaleHallucinations and Evaluation in Multimodal RAGDetecting Multimodal HallucinationsEvaluating Multimodal Retrieval and GenerationConclusion
9. Knowledge-Enhanced RAG
Knowledge Graphs: An OverviewHow Do You Search a Knowledge Graph?Ontologies Versus SchemasUsing Knowledge Graphs in RAGBuilding a Knowledge Graph for MoviesUsing the Knowledge Graph at Query TimeChoosing Between Enrichment and Hybrid RetrievalBuilding Knowledge GraphsAutomating Knowledge Graph ConstructionLeveraging Standard Ontologies and Knowledge GraphsGraphRAGThe Graph Database InfrastructureGraph Update Patterns and EvolutionThe Accuracy/Cost Trade-offConclusion
10. The Future of RAG
The Evolution of RetrievalThe Shift to Agentic RAGThe Reality of Data Gravity and Federated RetrievalThe Impact of Longer ContextFrom Prompt Engineering to Context EngineeringThe Shift from Reactive to Proactive RAGRAG at the Edge: Small Language ModelsGovernance and Compliance at ScaleConclusion: The Living Knowledge Base
Index
About the Authors

Content preview from Hands-On RAG for Production

Chapter 6. Evaluating Your RAG Application

Whether you build your RAG on your own (DIY) or use a RAG platform, you need to be able to measure the quality of responses that users see when they use your RAG application. This is known as RAG evaluation, which measures how accurately the system finds the right documents or chunks (retrieval accuracy) and how coherently and correctly it crafts its response from those documents or chunks (generation accuracy).

Before we jump into the details, it is helpful to distinguish between the two types of RAG evaluation:

Offline evaluation: Performed during the development cycle. These are deep, often resource-heavy evaluations used to optimize your pipeline settings before deployment.
Online evaluation: Performed on live traffic. This identifies how real users interact with the system but requires a lightweight approach to maintain a low-latency user experience.

In this chapter, we focus primarily on offline RAG evaluation, discussing why it’s important, which metrics you should consider, and how to interpret each metric. Then, in “Online RAG Evaluation”, we briefly discuss online evaluation and provide some best practices.

How Does RAG Fail?

Not having a systematic approach to measuring the quality of your RAG application is not just a technical oversight; it’s a business risk that can undermine your entire AI strategy. Without a structured, rigorous evaluation framework for RAG, you risk deploying applications that produce hallucinations ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9798341621701Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Hands-On RAG for Production

by Ofer Mendelevitch, Forrest Sheng Bao

Chapter 6. Evaluating Your RAG Application

How Does RAG Fail?

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.