book

The Art of AI Product Development

by Janna Lipenkova

May 2025

Beginner to intermediate

368 pages

11h 44m

English

Manning Publications

Read now

Unlock full access

1.1 Building with AI: The reality check1.2 A retrospective1.3 The anatomy of an AI system1.3.1 Opportunity space1.3.2 Solution space1.4 Learning with this book

2.1 Sourcing AI ideas and opportunities2.1.1 The AI opportunity tree2.1.2 Sources of AI opportunities2.1.3 Vertical vs. horizontal opportunities2.1.4 Navigating different scenarios for AI integration2.2 Prioritizing AI opportunities2.2.1 Defining your prioritization criteria2.2.2 Deciding on the next opportunity2.2.3 Balancing quick wins vs. long-term investments2.3 Shaping your opportunities2.3.1 The careful approach: Ready, aim, fire2.3.2 The fast approach: Ready, fire, aim2.3.3 Comparing the careful and fast approaches
3.1 Data3.1.1 The modality of your data3.1.2 Unlabeled vs. labeled data3.2 Different types of intelligence3.2.1 Rule-based AI3.2.2 Machine learning3.3 User experience3.3.1 Basic types of AI interfaces3.3.2 Assisted, augmented, and autonomous intelligence
4.1 Unsupervised learning4.1.1 Using clustering for behavioral segmentation4.1.2 Preparing training data for clustering4.1.3 Selecting and training a clustering model4.1.4 Evaluating clustering models4.1.5 Optimizing the clustering algorithm4.1.6 Acting on clustering outputs4.2 Supervised learning4.2.1 Preparing training data for classification4.2.2 Selecting and training a classification model4.2.3 Evaluating and optimizing the classification model4.2.4 Acting on classification outputs4.3 Time series and trend analysis4.3.1 Adding the time dimension to your data4.3.2 Extracting meaning from time series data4.3.3 Acting on time series insights4.4 Personalized recommendations4.4.1 Types of recommendation algorithms4.4.2 Evaluating and optimizing the recommendations
5.1 How language models work5.1.1 Understanding the training data of a language model5.1.2 The task of language modeling5.1.3 Expanding the capabilities of a language model5.2 Usage scenarios for language models5.2.1 Direct interaction between user and model5.2.2 Programmatic use5.2.3 Using the language model for predefined tasks5.3 Mapping the language model landscape5.3.1 Mainstream commercial LLMs5.3.2 Open source models5.3.3 Reasoning language models5.3.4 Small language models5.3.5 Multimodal models5.4 Managing the language model lifecycle5.4.1 Model selection5.4.2 Evaluating language models5.4.3 Customizing language models to your requirements5.4.4 Collecting feedback during production5.4.5 Continuously optimizing your language model setup
6.1 Basics of prompt engineering6.1.1 Zero-shot prompting6.1.2 Structuring your prompt engineering with prompt components and templates6.2 Few-shot prompting: Learning by analogy6.2.1 Basics of few-shot prompting6.2.2 Automating few-shot prompting6.3 Injecting reasoning into language models6.3.1 Chain-of-thought6.3.2 Self-consistency6.3.3 Reflection and iterative improvement6.4 Best practices for prompt engineering6.4.1 General guidelines6.4.2 Systematizing the prompt engineering process
7.1 Specializing your language model with custom data7.1.1 How prompt engineering falls short over time7.1.2 Summarizing the interview7.2 Retrieving relevant documents with semantic search7.2.1 The role of search in the B2B context7.2.2 Searching with semantic embeddings7.2.3 Evaluating search7.2.4 Optimizing your search system7.3 Building an end-to-end RAG system7.3.1 A basic RAG setup7.3.2 Evaluating your RAG system7.3.3 Optimizing your RAG system
8.1 Uncovering opportunities for fine-tuning8.1.1 Alex’s customer interview8.1.2 Evaluating fine-tuning as a solution8.2 Fine-tuning language models for different objectives8.2.1 Domain-specific fine-tuning8.2.2 Supervised fine-tuning8.2.3 Instruction fine-tuning
9.1 Providing language models with access to external tools9.1.1 Categories of tools9.1.2 Turning the human into a tool9.1.3 The ecosystem of tools9.1.4 Integrating tools with a language model9.2 Assembling the agent system9.2.1 The language model as the brain of the agent9.2.2 Planning the task execution9.2.3 Learning from memory9.3 Building at the frontier of AI agents9.3.1 Common challenges of agent systems9.3.2 Overcoming the limitations of agent systems9.4 Trends and opportunities for AI agents9.4.1 Scaling up with multi-agent collaboration9.4.2 Chatting with your data9.4.3 Autonomous enterprise
10.1 Discovery and user research10.1.1 Identifying the best opportunities for automation and augmentation10.1.2 Understanding the skills and psychology of your users10.1.3 Validating AI design concepts10.2 Designing the UI10.2.1 An initial user journey10.2.2 Guidelines and patterns for AI UX design10.3 Collecting feedback and co-creating with your users10.3.1 Types of user feedback10.3.2 Activating your users to provide feedback
11.1 Security: Protecting sensitive assets11.1.1 Data security11.1.2 Model security11.1.3 Usage security11.2 Privacy: Maintaining trust through transparency11.2.1 Managing privacy in the context of generative AI11.2.2 Incorporating privacy-by-design11.2.3 Regulatory context11.3 Mitigating bias in AI systems11.3.1 Training data bias11.3.2 Algorithmic bias11.3.3 Feedback loop bias11.3.4 Regulatory context11.4 Providing transparency11.4.1 Explainability: Showing how AI makes decisions11.4.2 Interpretability: Making AI outputs intuitive and accessible11.4.3 Accountability and oversight: Managing responsibility in AI decisions11.5 A proactive approach to AI governance
12.1 Efficient cross-functional collaboration in the AI team12.1.1 Building an AI team12.1.2 Data science and AI development12.1.3 Software engineering12.1.4 User experience design12.1.5 Domain expertise12.1.6 Troubleshooting collaboration challenges12.2 Getting buy-in from business stakeholders12.2.1 Executives12.2.2 Sales and marketing teams12.2.3 Customer success teams12.2.4 Compliance and legal departments12.3 Communicating with customers and users12.3.1 Communicating the value of your AI12.3.2 Communicating about AI failure12.3.3 Addressing the concerns of your users12.3.4 Educating about the right usage of your AI system12.3.5 Turning users into co-creators12.3.6 Differentiating between B2B and B2C contexts
A.1 How to use this appendixA.2 Chapter 2: Discovering and prioritizing AI opportunitiesA.2.1 Sourcing AI ideasA.2.2 Identifying AI-friendly problemsA.3 Chapter 3: Mapping the AI solution spaceA.3.1 Identifying data modalitiesA.3.2 Supervised vs. unsupervised learningA.3.3 Selecting the correct AI approach and interfaceA.4 Chapter 4: Predictive AIA.5 Chapter 5: Exploring and evaluating language modelsA.5.1 Selecting an LLMA.5.2 LM customization and optimizationA.6 Chapter 6: Prompt engineeringA.6.1 Structured promptsA.6.2 Selecting a prompting techniqueA.6.3 Systematizing prompt engineeringA.6.4 Evaluating prompt performanceA.6.5 Troubleshooting common issues in few-shot promptsA.6.6 Template for documenting prompt experimentsA.7 Chapter 7: Search and retrieval-augmented generationA.7.1 RAG and document retrievalA.7.2 Ensuring optimal RAG performanceA.8 Chapter 8: Fine-tuning language modelsA.8.1 Checklist for creating fine-tuning dataA.8.2 Tools and techniques for preparing your fine-tuning dataA.9 Chapter 9: Automating workflows with agentic AIA.9.1 Assessment and evaluation of automated workflowsA.9.2 Template: Agent workflow automation trackingA.10 Chapter 10: AI user experience: Designing for uncertaintyA.10.1 Working with AI uncertaintyA.10.2 Structuring AI-driven user feedback loopsA.10.3 Recipe for partial explanations of your AI systemA.11 Chapter 11: AI governanceA.11.1 AI security measuresA.11.2 Privacy compliance and privacy-by-designA.11.3 Bias detection and fairness in AIA.11.4 AI transparency and accountabilityA.12 Chapter 12: Working with your stakeholdersA.12.1 Best practices for stakeholder communicationA.12.2 AI communication strategy for AI failuresA.12.3 Communicating AI initiatives to business stakeholdersA.12.4 AI adoption and user educationA.12.5 Development timeline templates for AI projects

Content preview from The Art of AI Product Development

8 Fine-tuning language models

This chapter covers

Why you might need to fine-tune language models
The product manager’s role in the fine-tuning process
Creating data for fine-tuning
Domain, supervised, and instruction fine-tuning

In the previous two chapters, you learned about prompt engineering and retrieval-augmented generation (RAG)—two powerful techniques to supply a language model (LM) with specialized knowledge during inference. However, if your app requires expert-level LM performance, you might soon hit the limits of these techniques. Prompt engineering will quickly start to feel like consulting a high-school graduate who possesses solid general knowledge and can converse across many topics, but struggles with highly specialized ...