An Illustrated Guide to AI Agents

Chapter 9. Multi-Modal Understanding

Agentic systems have great potential when developed for real-world applications. These applications, however, don’t limit themselves to only text-based information, as is the case with generic LLMs. An agent may need to process a video, process an image to judge a website design, or be able to listen to your spoken request.

This understanding of different types of information, not just written text, but also images, audio, and videos, is called multi-modal understanding. When you watch a video and read the caption, your brain combines all those signals into one clear understanding.

Multi-modal LLMs are designed to do something similar: take in multiple kinds of input (different modalities) and process them together to form a more accurate response. An agent that can only read or write text will miss important parts of what’s going on in everyday situations, such as developing the UI of an application or an agentic browser needing to process images and texts of websites.

Some multi-modal LLMs are not only able to process different modalities but also output them. Imagine a system where the LLM is not only able to output text, but also audio or images. Such a model could use their “voice” to speak to you or to generate stock images for your website.

Multi-modal LLMs take an important place in single- and multi-agent systems (MASs). Since they can process all kinds of information, they are often the main “brain” behind these systems and are more ...