book

Effective Conversational AI

Name: Effective Conversational AI
ISBN: 9781633436404

by Cari Jacobs, Andrew Freed, Eniko Rozsa

April 2025

Intermediate to advanced

328 pages

10h 22m

English

Manning Publications

Audiobook available

Read now

Unlock full access

Effective Conversational AI: Chatbots that work
copyright
contents
dedication
foreword
preface
acknowledgments
about this book
about the authors
about the cover illustration

Part 1 Framework for improving conversational AI
1 What makes conversational AI work?
1.1 Introduction to conversational AI1.1.1 Why use conversational AI?1.1.2 How does conversational AI work?1.1.3 How you build conversational AI1.2 Introduction to generative AI in conversational AI1.2.1 What is generative AI1.2.2 Generative AI guardrails1.2.3 Effectively using generative AI in conversational AI1.3 Introducing continuous improvement in conversational AI1.3.1 Why continuously improve1.3.2 The continuous improvement cycle1.3.3 Communicating continuous improvement to stakeholders1.4 Follow along
2 Building a conversational AI
2.1 Building an FAQ bot2.1.1 FAQ bot foundations2.1.2 Static question and answering2.1.3 Dynamic question and answering2.2 Routing agents and process-oriented bots2.2.1 Routing agents2.2.2 Transitioning from a routing agent to a process-oriented bot2.3 Responding to the user with generative AI2.3.1 Integrating with an LLM2.3.2 Routing requests to an LLM
3 Planning for improvement
3.1 Knowing when you need to improve3.2 Your cross-functional team3.3 Driving to the same goal3.3.1 Revisit business goals3.3.2 Effectiveness3.3.3 Coverage3.4 Identifying and resolving problems3.4.1 Finding problems3.4.2 Group review3.4.3 Determining acceptance criteria3.5 Developing and delivering fixes3.5.1 Sprint planning3.5.2 Measure again
Part 2 Pattern: AI doesn’t understand
4 Understanding what your users really want
4.1 Fundamentals of understanding4.1.1 The impact of weak understanding4.1.2 What causes weak understanding?4.1.3 How do we achieve understanding with traditional conversational AI?4.1.4 How do we achieve understanding with generative AI?4.2 How is understanding measured?4.2.1 Measuring understanding for traditional (classification-based) AI4.2.2 Measuring understanding for generative AI4.2.3 Measuring understanding with direct user feedback4.3 Assessing where you are today4.3.1 Assessing your traditional (classification-based) AI solution4.3.2 Assessing your generative AI solution4.4 Obtaining and preparing test data from logs4.4.1 Obtaining production logs4.4.2 Guidelines for identifying candidate test utterances4.4.3 Preparing and scrubbing data for use in iterative improvements4.4.4 The annotation process4.5 What does the data tell us?4.5.1 Interpreting annotated logs for traditional (classification-based) AI4.5.2 Interpreting annotated logs for generative AI4.5.3 The case for iterative improvement
5 Improving weak understanding for traditional AI
5.1 Building your improvement plan5.1.1 Identify problematic patterns in misunderstood utterances5.1.2 Incremental improvements5.1.3 Where to start: Identifying the biggest problems5.2 Solving “wrong intent matched”5.2.1 Improve recall for one intent5.2.2 Improve precision for one intent5.2.3 Improve the F1 score for one intent5.2.4 Improve precision and recall for multiple intents5.3 Solving “no intent matched”5.3.1 Clustering utterances for new intents5.3.2 When to stop adding intents5.4 Supplementing traditional AI with generative content5.4.1 Combining traditional and generative AI for an intent5.4.2 Prompting to convey understanding
6 Enhancing responses with retrieval-augmented generation
6.1 Beyond intents: The role of search in conversational AI6.1.1 Using search in conversational AI6.1.2 Benefits of traditional search6.1.3 Drawbacks of traditional search6.2 Beyond search: Generating answers with RAG6.2.1 Using RAG in conversational AI6.2.2 Benefits of RAG6.2.3 Combining RAG with other generative AI use cases6.2.4 Comparing intents, search, and RAG approaches6.3 How is RAG implemented?6.3.1 High-level implementation6.3.2 Preparing your document repository for RAG6.4 Additional considerations of RAG implementations6.4.1 Can’t we just use an LLM directly?6.4.2 Keeping answers current and relevant with RAG6.4.3 How easy is it to set up the ingestion pipeline?6.4.4 Handling latency6.4.5 When to use a fallback mechanism and when to search6.5 Evaluating and analyzing RAG performance6.5.1 Indexing metrics6.5.2 Retrieval metrics6.5.3 Generation metrics6.5.4 Comparing efficiency of indexing and embedding solutions for RAG
7 Augmenting intent data with generative AI
7.1 Getting started7.1.1 Why do it: Pros and cons7.1.2 What you need7.1.3 How to use the augmented data7.2 Hardening your existing intents7.2.1 Get creative with synonyms7.2.2 Generate new grammatical variations7.2.3 Build strong intents from LLM output7.2.4 Creating even more examples with templates7.3 Getting more creative7.3.1 Brainstorm additional intents7.3.2 Check for confusion
Part 3 Pattern: AI is too complex
8 Streamlining complex flows
8.1 The pain of complexity8.1.1 Complexity’s effect on the end user8.1.2 Complexity’s effect on business metrics8.1.3 The incremental cost and benefit of reducing complexity for the user8.2 Simplifying and streamlining the user journey8.2.1 Spotting complex dialogue flows8.2.2 Using what is known about the user8.2.3 Aligning with the user’s mental model8.2.4 Allowing flexibility in the expected user responses8.2.5 Supporting self-service task flows with API/backend processes
9 Harnessing context for an adaptive virtual assistant experience
9.1 Importance of context in virtual assistant performance9.1.1 How context influences user interactions9.1.2 What is contextual information?9.2 Understanding modality9.2.1 Comparing modalities9.2.2 Importance of modality in designing virtual assistant flows9.2.3 Examples of how modality affects user experience9.2.4 Voice bot design considerations9.3 Enhancing context awareness and improving the overall user experience with RAG9.3.1 Designing adaptive flows with RAG9.3.2 Strategies for retrieving and generating contextually relevant responses9.3.3 Maintaining and updating adaptive flows
10 Reducing complexity with generative AI
10.1 AI-assisted process flows at build time10.1.1 Generating dialogue flows with generative AI10.1.2 Improving dialogue flow with generative AI10.2 AI-assisted process flows at run time10.2.1 Executing dialogue flows with generative AI10.2.2 Using LLM for a search process10.3 AI-assisted flows at test time10.3.1 Setting up generative AI to be the user10.3.2 Setting up the conversational test
Part 4 Pattern: Reduce friction
11 Reducing opt-outs
11.1 What drives opt-out behavior?11.1.1 Immediate opt-out drivers11.1.2 Motivations for later opt-outs11.1.3 Gathering data on opt-out behavior11.2 Reducing immediate opt-outs11.2.1 Start with a great experience: Greetings and introductions11.2.2 Convey capabilities and set expectations11.2.3 Incentivize self-service11.2.4 Allow the user to opt in11.3 Reducing other opt-outs11.3.1 Try hard to understand11.3.2 Try hard to be understood11.3.3 Be flexible and accommodating11.3.4 Convey progress11.3.5 Anticipate additional user needs11.3.6 Don’t be rude11.4 Opt-out retention11.4.1 Start right away by collecting opt-out data11.4.2 Implementing an opt-out retention flow11.5 Improving dialogue with generative AI11.5.1 Improving error messages with generative AI11.5.2 Improving greeting messages with generative AI11.6 Sometimes it’s okay to escalate
12 Conversational summarization for smooth handoff
12.1 Intro to summarization12.1.1 Why summarization is needed12.1.2 Elements of effective summaries12.2 Preparing your chatbot for summarization12.2.1 Using out-of-the-box elements12.2.2 Instrumenting your chatbot for transcripts12.2.3 Instrumenting your chatbot (for data points)12.3 Improving summaries with generative AI12.3.1 Generating a text summary of a transcript with summarizing prompts12.3.2 Generating a structured summary of a transcript with extractive prompts

Content preview from Effective Conversational AI

4 Understanding what your users really want

This chapter covers

Recognizing indicators of weak understanding
Measuring chatbot understanding
Assessing your chatbot’s current state
Collecting and preparing log data to measure chatbot understanding
Interpreting initial log data

A good chatbot experience is generally associated with the chatbot identifying (understanding) what the user wants. This is one of the key metrics you will use to measure performance. Sometimes a chatbot is deployed and has great initial understanding (or at least “good enough” for a pilot program). Over time, though, you may notice that it is returning wrong answers. Maybe your users are complaining more, either directly to the chatbot (“That doesn’t answer my question!”) ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781633436404Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Effective Conversational AI

by Cari Jacobs, Andrew Freed, Eniko Rozsa