Building Live Voice Agents

Chapter 2. Understanding Multimodality: Beyond Text

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be Chapter 2 of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

In Chapter 1, we defined an AI agent as an autonomous system that perceives its environment, makes decisions, and uses tools to achieve a specific goal. For most software agents, perception has historically meant perception of text: a prompt, a chat message, a transcript, a document, or a structured payload. Text is efficient, but it is also a compressed representation of the world. When you ask a text-only model to help debug a user interface, you have to translate what you see into words. When you want help with a strange sound in a machine, you have to describe the sound rather than provide the sound itself.

Tone, timing, gesture, layout, visual detail, and background sound can all matter to the task, but they are hard to capture in a typed prompt. In effect, this text bottleneck forces the user or application developer to act as the sensory translator for the agent. Some information survives that translation, but much of it does not.

This chapter ...