An Illustrated Guide to AI Agents

Chapter 9. Multi-Modal Understanding

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 9th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

Agentic systems have great potential when developed for real-world applications. These applications, however, don’t limit themselves to only text-based information, as is the case with generic LLMs. An agent may need to process a video, process an image to judge a website design, or be able to listen to your spoken request.

This understanding of different types of information, not just written text, but also images, audio, and videos, is called multi-modal understanding. When you watch a video and read the caption, your brain combines all those signals into one clear understanding.

Multi-modal LLMs are designed to do something similar: take in multiple kinds of input (different modalities) and process them together to form a more accurate response. An agent that can only read or write text will miss important parts of what’s going on in everyday situations, such as developing the UI of an application or an agentic browser needing to process images and texts ...