book

Effective Conversational AI

by Eniko Rozsa, Andrew Freed, Cari Jacobs

April 2025

Intermediate to advanced

328 pages

10h 22m

English

Manning Publications

Read now

Unlock full access

1.1 Introduction to conversational AI1.1.1 Why use conversational AI?1.1.2 How does conversational AI work?1.1.3 How you build conversational AI1.2 Introduction to generative AI in conversational AI1.2.1 What is generative AI1.2.2 Generative AI guardrails1.2.3 Effectively using generative AI in conversational AI1.3 Introducing continuous improvement in conversational AI1.3.1 Why continuously improve1.3.2 The continuous improvement cycle1.3.3 Communicating continuous improvement to stakeholders1.4 Follow along
2.1 Building an FAQ bot2.1.1 FAQ bot foundations2.1.2 Static question and answering2.1.3 Dynamic question and answering2.2 Routing agents and process-oriented bots2.2.1 Routing agents2.2.2 Transitioning from a routing agent to a process-oriented bot2.3 Responding to the user with generative AI2.3.1 Integrating with an LLM2.3.2 Routing requests to an LLM
3.1 Knowing when you need to improve3.2 Your cross-functional team3.3 Driving to the same goal3.3.1 Revisit business goals3.3.2 Effectiveness3.3.3 Coverage3.4 Identifying and resolving problems3.4.1 Finding problems3.4.2 Group review3.4.3 Determining acceptance criteria3.5 Developing and delivering fixes3.5.1 Sprint planning3.5.2 Measure again
4.1 Fundamentals of understanding4.1.1 The impact of weak understanding4.1.2 What causes weak understanding?4.1.3 How do we achieve understanding with traditional conversational AI?4.1.4 How do we achieve understanding with generative AI?4.2 How is understanding measured?4.2.1 Measuring understanding for traditional (classification-based) AI4.2.2 Measuring understanding for generative AI4.2.3 Measuring understanding with direct user feedback4.3 Assessing where you are today4.3.1 Assessing your traditional (classification-based) AI solution4.3.2 Assessing your generative AI solution4.4 Obtaining and preparing test data from logs4.4.1 Obtaining production logs4.4.2 Guidelines for identifying candidate test utterances4.4.3 Preparing and scrubbing data for use in iterative improvements4.4.4 The annotation process4.5 What does the data tell us?4.5.1 Interpreting annotated logs for traditional (classification-based) AI4.5.2 Interpreting annotated logs for generative AI4.5.3 The case for iterative improvement
5.1 Building your improvement plan5.1.1 Identify problematic patterns in misunderstood utterances5.1.2 Incremental improvements5.1.3 Where to start: Identifying the biggest problems5.2 Solving “wrong intent matched”5.2.1 Improve recall for one intent5.2.2 Improve precision for one intent5.2.3 Improve the F1 score for one intent5.2.4 Improve precision and recall for multiple intents5.3 Solving “no intent matched”5.3.1 Clustering utterances for new intents5.3.2 When to stop adding intents5.4 Supplementing traditional AI with generative content5.4.1 Combining traditional and generative AI for an intent5.4.2 Prompting to convey understanding
6.1 Beyond intents: The role of search in conversational AI6.1.1 Using search in conversational AI6.1.2 Benefits of traditional search6.1.3 Drawbacks of traditional search6.2 Beyond search: Generating answers with RAG6.2.1 Using RAG in conversational AI6.2.2 Benefits of RAG6.2.3 Combining RAG with other generative AI use cases6.2.4 Comparing intents, search, and RAG approaches6.3 How is RAG implemented?6.3.1 High-level implementation6.3.2 Preparing your document repository for RAG6.4 Additional considerations of RAG implementations6.4.1 Can’t we just use an LLM directly?6.4.2 Keeping answers current and relevant with RAG6.4.3 How easy is it to set up the ingestion pipeline?6.4.4 Handling latency6.4.5 When to use a fallback mechanism and when to search6.5 Evaluating and analyzing RAG performance6.5.1 Indexing metrics6.5.2 Retrieval metrics6.5.3 Generation metrics6.5.4 Comparing efficiency of indexing and embedding solutions for RAG
7.1 Getting started7.1.1 Why do it: Pros and cons7.1.2 What you need7.1.3 How to use the augmented data7.2 Hardening your existing intents7.2.1 Get creative with synonyms7.2.2 Generate new grammatical variations7.2.3 Build strong intents from LLM output7.2.4 Creating even more examples with templates7.3 Getting more creative7.3.1 Brainstorm additional intents7.3.2 Check for confusion
8.1 The pain of complexity8.1.1 Complexity’s effect on the end user8.1.2 Complexity’s effect on business metrics8.1.3 The incremental cost and benefit of reducing complexity for the user8.2 Simplifying and streamlining the user journey8.2.1 Spotting complex dialogue flows8.2.2 Using what is known about the user8.2.3 Aligning with the user’s mental model8.2.4 Allowing flexibility in the expected user responses8.2.5 Supporting self-service task flows with API/backend processes
9.1 Importance of context in virtual assistant performance9.1.1 How context influences user interactions9.1.2 What is contextual information?9.2 Understanding modality9.2.1 Comparing modalities9.2.2 Importance of modality in designing virtual assistant flows9.2.3 Examples of how modality affects user experience9.2.4 Voice bot design considerations9.3 Enhancing context awareness and improving the overall user experience with RAG9.3.1 Designing adaptive flows with RAG9.3.2 Strategies for retrieving and generating contextually relevant responses9.3.3 Maintaining and updating adaptive flows
10.1 AI-assisted process flows at build time10.1.1 Generating dialogue flows with generative AI10.1.2 Improving dialogue flow with generative AI10.2 AI-assisted process flows at run time10.2.1 Executing dialogue flows with generative AI10.2.2 Using LLM for a search process10.3 AI-assisted flows at test time10.3.1 Setting up generative AI to be the user10.3.2 Setting up the conversational test
11.1 What drives opt-out behavior?11.1.1 Immediate opt-out drivers11.1.2 Motivations for later opt-outs11.1.3 Gathering data on opt-out behavior11.2 Reducing immediate opt-outs11.2.1 Start with a great experience: Greetings and introductions11.2.2 Convey capabilities and set expectations11.2.3 Incentivize self-service11.2.4 Allow the user to opt in11.3 Reducing other opt-outs11.3.1 Try hard to understand11.3.2 Try hard to be understood11.3.3 Be flexible and accommodating11.3.4 Convey progress11.3.5 Anticipate additional user needs11.3.6 Don’t be rude11.4 Opt-out retention11.4.1 Start right away by collecting opt-out data11.4.2 Implementing an opt-out retention flow11.5 Improving dialogue with generative AI11.5.1 Improving error messages with generative AI11.5.2 Improving greeting messages with generative AI11.6 Sometimes it’s okay to escalate
12.1 Intro to summarization12.1.1 Why summarization is needed12.1.2 Elements of effective summaries12.2 Preparing your chatbot for summarization12.2.1 Using out-of-the-box elements12.2.2 Instrumenting your chatbot for transcripts12.2.3 Instrumenting your chatbot (for data points)12.3 Improving summaries with generative AI12.3.1 Generating a text summary of a transcript with summarizing prompts12.3.2 Generating a structured summary of a transcript with extractive prompts

Overview

Create and improve conversational AI with the latest patterns, best practices, and tools, including generative AI models.

Conversational AI (CAI) tools are built to solve problems, but all-too-often they just end up causing pain for users–and developers! Effective Conversational AI reveals best practices and industry-tested techniques for creating chatbots and conversational AI tools that are reliable at an enterprise scale. With the tested ideas and examples in this book, you’ll learn to build chatbots that your customers and colleagues will actually want to use!

In Effective Conversational AI you’ll learn how to:

Create high-quality chatbots and other conversational AI experiences
Plan for continuous improvement
Incorporate generative AI solutions to improve quality, accuracy, and usability
Evaluate user experience and business results

Effective Conversational AI introduces continuous improvement practices that are vital for the constant betterment and evolution of chatbots and CAI tools. It introduces the three most-common forms of chatbot—Q&A, process-oriented, and routing agents—and presents a reliable framework for continuously improving each one. Using modern generative AI and tried-and-tested classic approaches, you’ll learn to deliver high performance chatbots that can guide a customer through complex end-to-end tasks—no human required!

About the Technology
Powerful new chatbot frameworks and Generative AI models can practically eliminate problems like misinterpreting user intent and delivering nonsensical answers. In this book, you’ll learn how to build chatbots that take advantage of large language models and other modern tools and create conversational AI experiences users will love.

About the Book
Effective Conversational AI teaches you how to build great chatbots that perform reliably even at enterprise scale. In it, you’ll learn how to clarify user intent using LLMs, respond accurately to unanticipated input, and use Retrieval Augmented Generation to keep responses up to date. Along the way, you’ll discover how to establish a feedback loop for continuous quality improvement and master techniques to integrate GenAI safely into conventional chatbot designs.

What's Inside

Blend Generative AI and conventional chatbot tools
Use LLMs to improve quality, accuracy, and usability
Plan for continuous improvement
Domain-specific responses using RAG

About the Reader
For developers, engineers, and product managers working with conversational AI.

About the Authors
Andrew Freed, Cari Jacobs, and Enikő Rózsa are seasoned conversational AI developers with IBM.

Quotes
A wonderful comprehensive guide written by individuals who have walked the AI conversational implementation journey into production.
- Sara Hines, AI innovation pioneer

Cuts through the hype and focuses on what really matters.
- Jerry Cuomo, IBM

A blueprint for building and measuring effective conversational AI systems.
- Corville Allen, Google

An invaluable resource for anyone looking to drive success with AI-driven conversations.
- Marc Nehme, Microsoft