Hands-On RAG for Production

Chapter 8. Multimodal RAG

So far we’ve mostly focused on text-based RAG, where the knowledge-grounding responses were confined to what could be written, ignoring the knowledge represented in other formats, including tables, images, audio, and video.

In reality, enterprise knowledge is available in many modalities—a profit and loss (P&L) table in a financial report, the visual instructions in an operator manual, or the spoken nuance in a customer service call. Without the ability to see the chart, hear the call, or read the table, the RAG system’s accuracy suffers: it will provide high-quality answers grounded in text-based documents, but provide low-quality or hallucinated responses when the information required for an accurate answer is inside a table or an image.

In this chapter, we explore multimodal RAG, and how to integrate data that comes in other (non-text) modalities into RAG. We will dive into the core strategies for integrating these other modalities, and examine the production challenges they bring.

To navigate this landscape, it is helpful to clarify what “multimodal” looks like in a production environment. While the dream is a single, “native” multimodal model that consumes raw audio and video as easily as text, the reality is often more complex. In practice, most current approaches to enterprise multimodal RAG fall into one of two categories:

The “conversion” approach

Using specialized parsers, automatic speech recognition (ASR), or vision–language models (VLMs) ...