book

Hands-On RAG for Production

by Ofer Mendelevitch, Forrest Sheng Bao

May 2026

Intermediate to advanced

358 pages

10h 52m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Includes

Quizzes

Foreword by Sharon Zhou
Foreword by Jim Dowling
Preface
What This Book Is AboutWho This Book Is ForWho This Book Is Not ForUsing Code ExamplesPrerequisitesNavigating This BookConventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
1. Introduction to Retrieval-Augmented Generation (RAG)
How Does RAG Work?The Blueprint of a RAG StackThe Ingestion FlowThe Query FlowExample: RAG with LangChainRAG Versus Other ApproachesRAG Versus “Chat with PDF”RAG Versus Fine-TuningKey Benefits of RAGRAG Is Scalable and EfficientRAG Helps Reduce HallucinationsRAG Enables ExplainabilityNear-Instant Addition and Removal of KnowledgeAccess Controls and SecurityRAG Use CasesVirtual Assistants and AI ChatbotsEnterprise Knowledge Management and Internal SearchAutomated Content Creation and Document SummarizationGenerating Attractive and Effective Personalized AdsQuestion-Answering SystemsMedical and Healthcare ApplicationsLegal and Compliance ResearchAdvanced RAGAgentic RAGMultimodal RAGRAG with Knowledge GraphsConclusion
2. The Base RAG Stack
RAG Stack FlowsThe Ingestion FlowThe Query FlowDocument ParsingExtracting Text from Various File Formats Document Parsing with Vision–Language ModelsCode Example: Parsing FilesText ChunkingChunking StrategiesCode Example: Chunking in PythonEmbedding ModelsWhat Is (an) Embedding?Selection Criteria for Embedding ModelsPractical Tips and ConsiderationsCode Example: Generating Embeddings with Sentence TransformersVector Databases and Vector SearchUnderstanding Vector-Based Similarity SearchApproximate Nearest Neighbor AlgorithmsVector DatabasesParameters to Consider When Using Vector SearchCode Example: Storing and Retrieving Vectors Using pgvectorGenerative LLMsLLMsRAG Prompt EngineeringEvaluating LLMs and Prompt TemplatesCode Example: Using Anthropic Claude to Generate Responses in RAGConclusion
3. Scaling Your RAG Stack
RAG at ScaleVolume and Complexity of DocumentsIndex FreshnessCost Management and OptimizationAdvanced Data IngestionHandling a Large Volume of DocumentsDealing with Inconsistent Data QualityHandling Large DocumentsExample: Splitting a Large PDF FileManaging Document Updates and RefreshAdvanced RetrievalThe Two-Stage Retrieval PipelineHybrid SearchRerankingImplementing GuardrailsGuardrails for AI SafetyPreventing Prompt Injection AttacksControlling Hallucinations in RAGDefining Hallucinations in RAGLLM Hallucinations Versus RAG HallucinationsHallucination DetectionHallucination CorrectionBuilding a Great RAG User ExperienceConsiderations for RAG User ExperienceMultimodal User InterfacesTools and Reference ImplementationsConclusion
4. Deploying RAG to Production
Challenges with RAG in ProductionResponse Quality and Reduced HallucinationsHigh LatencyData Security and PrivacyVendor Chaos and Integration WoesTeam and ExpertiseTotal Cost of OwnershipRAG EvaluationA Reference Production ArchitectureSuccessful Transition from Proof of Concept to ProductionSummarize What You Learned in the Proof of ConceptDefine Goals and RequirementsEnsuring Continued RAG SuccessConclusion
5. The RAG Platform
DIY Versus Platform RAGCore RAG CapabilitiesData SourcesRAG Sprawl and Centralized GovernanceCost and UpkeepDeployment OptionsExample RAG Platform: VectaraGetting StartedIngesting Data into VectaraRunning QueriesHallucination CorrectionOther RAG Admin API EndpointsConclusion
6. Evaluating Your RAG Application
How Does RAG Fail?Retrieval FailuresGeneration FailuresFailures Due to Inadequate Data IngestionSummary of RAG FailuresUsing LLMs for Evaluation: LLM-as-a-JudgeWhat Is LLM-as-a-Judge?How LLM-as-a-Judge WorksRAG Evaluation MetricsRetrieval MetricsGeneration MetricsBias and SafetyRAG Evaluation OfferingsOpen RAG EvalRetrieval-Augmented Generation AssessmentDeepEvalAmazon BedrockHuman FeedbackIntegrating RAG Evaluation in ProductionUsing LLM-as-a-Judge in ProductionOffline RAG EvaluationOnline RAG EvaluationSystem Metrics: Latency and UptimeLatency and ThroughputReliability and UptimeCost and Resource EfficiencyConclusion
7. From RAG to AI Agents
What Is an AI Agent?The Agentic StackSingle-Agent Versus Multi-Agent SystemsAgentic Use CasesAgents in Customer ServiceAgents in Financial ServicesAgentic AI in HealthcareAI Coding AgentsThe Agentic LoopTool CallingModel Context ProtocolModel Context Protocol ArchitectureMCP in Enterprise Agentic AIAgent-to-Agent CommunicationHands-On with Agentic AI FrameworksAI Chatbots Using LangChainDocument Generation Agent with LlamaIndexBuilding an Agent with VectaraBuilding a Multi-Agent System with CrewAIAgentic MemoryShort-Term Versus Long-Term MemoryImplementation Memory with Agentic RAGEnterprise Guardrails: Privacy and IntegrityEvaluation and Observability with AI AgentsAgentic ObservabilityTracing an AgentAgentic Observability MetricsTools for Agentic ObservabilityConclusion

8. Multimodal RAG
Documents with Embedded TablesWhy Are Embedded Tables Important?Extracting Tables from DocumentsWhy Naive Chunking Fails for TablesProcessing Tables for RAGDealing with Multi-Page TablesDocuments with Embedded ImagesThe Image Summarization ApproachMultimodal Retrieval with a Shared Embedding SpaceAudio and Video in RAGThe Baseline: High-Fidelity TranscriptionVisual Semantics: The “Red Button” ProblemProduction ConsiderationsComputational Economics and LatencyModality AlignmentThe Interface Layer: Visual CitationsSecurity, Privacy, and GovernanceDeep Observability, Tracing, and Security at ScaleHallucinations and Evaluation in Multimodal RAGDetecting Multimodal HallucinationsEvaluating Multimodal Retrieval and GenerationConclusion
9. Knowledge-Enhanced RAG
Knowledge Graphs: An OverviewHow Do You Search a Knowledge Graph?Ontologies Versus SchemasUsing Knowledge Graphs in RAGBuilding a Knowledge Graph for MoviesUsing the Knowledge Graph at Query TimeChoosing Between Enrichment and Hybrid RetrievalBuilding Knowledge GraphsAutomating Knowledge Graph ConstructionLeveraging Standard Ontologies and Knowledge GraphsGraphRAGThe Graph Database InfrastructureGraph Update Patterns and EvolutionThe Accuracy/Cost Trade-offConclusion
10. The Future of RAG
The Evolution of RetrievalThe Shift to Agentic RAGThe Reality of Data Gravity and Federated RetrievalThe Impact of Longer ContextFrom Prompt Engineering to Context EngineeringThe Shift from Reactive to Proactive RAGRAG at the Edge: Small Language ModelsGovernance and Compliance at ScaleConclusion: The Living Knowledge Base
Index
About the Authors

Content preview from Hands-On RAG for Production

Chapter 3. Scaling Your RAG Stack

In Chapter 2, you saw the basic components of a RAG stack: document parsing, chunking, embedding models, and vector search, as well as using an LLM for the final generation of the response to the user. With this knowledge, you should now be able to build end-to-end RAG applications that work pretty well on small to medium datasets, and experience for yourself how RAG works in practice.

In this chapter, we tackle more advanced techniques that help you bring your RAG stack to enterprise scale, without sacrificing latency or response quality, including data ingestion, advanced retrieval techniques, guardrails, and handling RAG hallucinations. Although not strictly part of scale, security and data privacy become important as you scale in production, and we cover those in Chapter 4 under “Data Security and Privacy”.

We end this chapter with a less commonly discussed but critical aspect of any RAG application: building a great user experience to make sure your frontend is as good as your backend.

RAG at Scale

When your RAG application grows in scale, things can become more complex relatively quickly. You had to deal with higher volumes of documents and queries, multiple document formats, integrating advanced retrieval mechanisms to maintain high-quality responses, hallucination mitigation, guardrails, and a lot more.

In this section, we’ll dive into the various challenges that come up with RAG at scale and how to address them.

Volume and Complexity ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9798341621701Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design