Multimodal, Real-Time AI Agent Systems

Chapter 2. Architecting for Real-Time AI Interaction

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 3rd chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

For decades, science fiction has set the expectations of our future with AI: a seamless, spoken conversation with an intelligent companion. We dreamed of the sophisticated computer from Star Trek or the helpful wit of Robby the Robot. But, for years, the actual experience with the first generation of voice assistants fell short of our expectations. We expected to be able to converse with computers to accomplish tasks. Instead, we got a glorified command-line experience. We were promised a conversation, but what we got was a rigid, unforgiving command line that forced us to learn its language.

This chapter is about finally building the future we expected.

We will begin by dissecting the limitations of the first generation and exposing their fundamental architectural flaw—the rigid, turn-based model—that made true conversation impossible. We will then explore the two technological pillars that shattered that old paradigm: the Transformer architecture ...