book

Hands-On RAG for Production

by Ofer Mendelevitch, Forrest Sheng Bao

May 2026

Intermediate to advanced

358 pages

10h 52m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Includes

Quizzes

Foreword by Sharon Zhou
Foreword by Jim Dowling
Preface
What This Book Is AboutWho This Book Is ForWho This Book Is Not ForUsing Code ExamplesPrerequisitesNavigating This BookConventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
1. Introduction to Retrieval-Augmented Generation (RAG)
How Does RAG Work?The Blueprint of a RAG StackThe Ingestion FlowThe Query FlowExample: RAG with LangChainRAG Versus Other ApproachesRAG Versus “Chat with PDF”RAG Versus Fine-TuningKey Benefits of RAGRAG Is Scalable and EfficientRAG Helps Reduce HallucinationsRAG Enables ExplainabilityNear-Instant Addition and Removal of KnowledgeAccess Controls and SecurityRAG Use CasesVirtual Assistants and AI ChatbotsEnterprise Knowledge Management and Internal SearchAutomated Content Creation and Document SummarizationGenerating Attractive and Effective Personalized AdsQuestion-Answering SystemsMedical and Healthcare ApplicationsLegal and Compliance ResearchAdvanced RAGAgentic RAGMultimodal RAGRAG with Knowledge GraphsConclusion
2. The Base RAG Stack
RAG Stack FlowsThe Ingestion FlowThe Query FlowDocument ParsingExtracting Text from Various File Formats Document Parsing with Vision–Language ModelsCode Example: Parsing FilesText ChunkingChunking StrategiesCode Example: Chunking in PythonEmbedding ModelsWhat Is (an) Embedding?Selection Criteria for Embedding ModelsPractical Tips and ConsiderationsCode Example: Generating Embeddings with Sentence TransformersVector Databases and Vector SearchUnderstanding Vector-Based Similarity SearchApproximate Nearest Neighbor AlgorithmsVector DatabasesParameters to Consider When Using Vector SearchCode Example: Storing and Retrieving Vectors Using pgvectorGenerative LLMsLLMsRAG Prompt EngineeringEvaluating LLMs and Prompt TemplatesCode Example: Using Anthropic Claude to Generate Responses in RAGConclusion
3. Scaling Your RAG Stack
RAG at ScaleVolume and Complexity of DocumentsIndex FreshnessCost Management and OptimizationAdvanced Data IngestionHandling a Large Volume of DocumentsDealing with Inconsistent Data QualityHandling Large DocumentsExample: Splitting a Large PDF FileManaging Document Updates and RefreshAdvanced RetrievalThe Two-Stage Retrieval PipelineHybrid SearchRerankingImplementing GuardrailsGuardrails for AI SafetyPreventing Prompt Injection AttacksControlling Hallucinations in RAGDefining Hallucinations in RAGLLM Hallucinations Versus RAG HallucinationsHallucination DetectionHallucination CorrectionBuilding a Great RAG User ExperienceConsiderations for RAG User ExperienceMultimodal User InterfacesTools and Reference ImplementationsConclusion
4. Deploying RAG to Production
Challenges with RAG in ProductionResponse Quality and Reduced HallucinationsHigh LatencyData Security and PrivacyVendor Chaos and Integration WoesTeam and ExpertiseTotal Cost of OwnershipRAG EvaluationA Reference Production ArchitectureSuccessful Transition from Proof of Concept to ProductionSummarize What You Learned in the Proof of ConceptDefine Goals and RequirementsEnsuring Continued RAG SuccessConclusion
5. The RAG Platform
DIY Versus Platform RAGCore RAG CapabilitiesData SourcesRAG Sprawl and Centralized GovernanceCost and UpkeepDeployment OptionsExample RAG Platform: VectaraGetting StartedIngesting Data into VectaraRunning QueriesHallucination CorrectionOther RAG Admin API EndpointsConclusion
6. Evaluating Your RAG Application
How Does RAG Fail?Retrieval FailuresGeneration FailuresFailures Due to Inadequate Data IngestionSummary of RAG FailuresUsing LLMs for Evaluation: LLM-as-a-JudgeWhat Is LLM-as-a-Judge?How LLM-as-a-Judge WorksRAG Evaluation MetricsRetrieval MetricsGeneration MetricsBias and SafetyRAG Evaluation OfferingsOpen RAG EvalRetrieval-Augmented Generation AssessmentDeepEvalAmazon BedrockHuman FeedbackIntegrating RAG Evaluation in ProductionUsing LLM-as-a-Judge in ProductionOffline RAG EvaluationOnline RAG EvaluationSystem Metrics: Latency and UptimeLatency and ThroughputReliability and UptimeCost and Resource EfficiencyConclusion
7. From RAG to AI Agents
What Is an AI Agent?The Agentic StackSingle-Agent Versus Multi-Agent SystemsAgentic Use CasesAgents in Customer ServiceAgents in Financial ServicesAgentic AI in HealthcareAI Coding AgentsThe Agentic LoopTool CallingModel Context ProtocolModel Context Protocol ArchitectureMCP in Enterprise Agentic AIAgent-to-Agent CommunicationHands-On with Agentic AI FrameworksAI Chatbots Using LangChainDocument Generation Agent with LlamaIndexBuilding an Agent with VectaraBuilding a Multi-Agent System with CrewAIAgentic MemoryShort-Term Versus Long-Term MemoryImplementation Memory with Agentic RAGEnterprise Guardrails: Privacy and IntegrityEvaluation and Observability with AI AgentsAgentic ObservabilityTracing an AgentAgentic Observability MetricsTools for Agentic ObservabilityConclusion

8. Multimodal RAG
Documents with Embedded TablesWhy Are Embedded Tables Important?Extracting Tables from DocumentsWhy Naive Chunking Fails for TablesProcessing Tables for RAGDealing with Multi-Page TablesDocuments with Embedded ImagesThe Image Summarization ApproachMultimodal Retrieval with a Shared Embedding SpaceAudio and Video in RAGThe Baseline: High-Fidelity TranscriptionVisual Semantics: The “Red Button” ProblemProduction ConsiderationsComputational Economics and LatencyModality AlignmentThe Interface Layer: Visual CitationsSecurity, Privacy, and GovernanceDeep Observability, Tracing, and Security at ScaleHallucinations and Evaluation in Multimodal RAGDetecting Multimodal HallucinationsEvaluating Multimodal Retrieval and GenerationConclusion
9. Knowledge-Enhanced RAG
Knowledge Graphs: An OverviewHow Do You Search a Knowledge Graph?Ontologies Versus SchemasUsing Knowledge Graphs in RAGBuilding a Knowledge Graph for MoviesUsing the Knowledge Graph at Query TimeChoosing Between Enrichment and Hybrid RetrievalBuilding Knowledge GraphsAutomating Knowledge Graph ConstructionLeveraging Standard Ontologies and Knowledge GraphsGraphRAGThe Graph Database InfrastructureGraph Update Patterns and EvolutionThe Accuracy/Cost Trade-offConclusion
10. The Future of RAG
The Evolution of RetrievalThe Shift to Agentic RAGThe Reality of Data Gravity and Federated RetrievalThe Impact of Longer ContextFrom Prompt Engineering to Context EngineeringThe Shift from Reactive to Proactive RAGRAG at the Edge: Small Language ModelsGovernance and Compliance at ScaleConclusion: The Living Knowledge Base
Index
About the Authors

Content preview from Hands-On RAG for Production

Chapter 9. Knowledge-Enhanced RAG

RAG has fundamentally changed how we approach information retrieval. We’ve moved beyond the rigid world of keyword matching into the more fluid and intuitive domain of semantic search. This shift allows us to find documents based on their conceptual meaning, not just the specific words they contain. In spite of its strengths, RAG systems built solely on vector search or hybrid search can struggle with queries that require understanding of relationships between entities in a more precise manner. This problem arises because vector search deals in probabilities and similarities, not deterministic facts. Here are a few examples:

Time-bound facts: Semantic search is not good at answering questions that depend on a specific point in time, because embeddings tend to blur together past and present information.¹; For example, a query like “Who was the CEO of Twitter in October 2022?” could return chunks that mention Elon Musk, Jack Dorsey, or Parag Agrawal, depending on what text is retrieved, without any clear alignment to the requested date. This often leads to either a bad response or the LLM trying to provide all the options without pinpointing the specific correct answer for 2022.
Intersection of multiple constraints: When a question requires satisfying more than one condition at once, semantic search tends to surface loosely related text rather than a guaranteed overlap.; Consider the query “Which drugs interact with both warfarin and grapefruit ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9798341621701Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Hands-On RAG for Production

by Ofer Mendelevitch, Forrest Sheng Bao

Chapter 9. Knowledge-Enhanced RAG

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.