AI Product Management

Chapter 3. Understanding AI Product Behavior

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 5th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

“Agents are just LLMs, Tools and you (the user) in a group chat. That’s all there is.”

Tal Raviv, after pretending to be an LLM in WhatsApp

Tal Raviv once performed an experiment at my AI Product Sense workshop that fundamentally changed how I think about large language models (LLMs). He announced he’d be “pretending to be Claude” in a WhatsApp group chat to help me plan a picnic. For the next ten minutes, he responded to student questions about the weather, data retrieval, and code generation. (Figure 3-1)

The audience couldn’t tell the difference. After the reveal, one student said: “Wait, that’s actually how it works? It’s just...messages?”. That’s the thing about LLMs. They’re simpler than most people think which makes their behavior even more alien to those used in traditional software.

This chapter builds the mental models you need to work with AI products effectively. We will explore how to handle non-determinism in probabilistic systems ...