book

Software Testing with Generative AI

Name: Software Testing with Generative AI
Author: Mark Winteringham
ISBN: 9781633437364

by Mark Winteringham

December 2024

Intermediate to advanced

304 pages

8h 57m

English

Manning Publications

Audiobook available

Read now

Unlock full access

Software Testing with Generative AI
Copyright
dedication
contents
front matter
forewordprefaceacknowledgmentsabout this bookWho should read this bookHow this book is organized: A road mapAbout the codeliveBook discussion forumabout the authorabout the cover illustration
Part 1. Mindset: Establishing a positive relationship with LLMs
1 Enhancing testing with large language models
1.1 Recognizing the effect of AI tools on testing and development1.1.1 Data generation1.1.2 Automated test building1.1.3 Test design1.2 Delivering value with LLMs1.2.1 A model for delivering value1.2.2 Using both human and AI abilities1.2.3 Being skeptical of LLMsSummary
2 Large language models and prompt engineering
2.1 LLMs explained2.2 Avoiding the risks of using LLMs2.2.1 Hallucinations2.2.2 Data provenance2.2.3 Data privacy2.3 Improving results with prompt engineering2.4 Examining the principles of prompt engineering2.4.1 Principle 1: Write clear and specific instructions2.4.2 Tactic 1: Use delimiters2.4.3 Tactic 2: Ask for structured output2.4.4 Tactic 3: Check for assumptions2.4.5 Tactic 4: Few-shot prompting2.4.6 Principle 2: Give the model time to “think”2.4.7 Tactic 1: Specify the steps to complete the task2.4.8 Tactic 2: Instruct the model to work out its own solution first2.5 Working with various LLMs2.5.1 Comparing LLMs2.5.2 Examining popular LLMs2.6 Creating a library of prompts2.7 Solving problems by using promptsSummary
3 Artificial intelligence, automation, and testing
3.1 The value of testing3.1.1 A different way of thinking about testing3.1.2 A holistic approach to testing3.2 How tools help with testing3.2.1 Automation bias3.2.2 Being selective with tooling3.3 Knowing when to use LLMs in testing3.3.1 Generative capabilities3.3.2 Transformation capabilities3.3.3 Enhancing capabilities3.3.4 LLMs in use in testingSummary
Part 2. Technique: Task identification and prompt engineering in testing

4 AI-assisted testing for developers
4.1 Examining the rise of the automated developer4.2 Pairing with LLMs4.2.1 Analyzing ideas4.2.2 Analyzing code4.2.3 Recognizing that a simulation is better than nothing at all4.3 Building in quality with AI assistance4.4 Creating our first TDD loop with LLMs4.4.1 Preparing the work4.4.2 Loop 1: Save a timesheet entry4.4.3 Loop 2: Retrieve a timesheet entry4.4.4 Loop 3: Calculating times for a project4.4.5 Refactoring code4.5 Improving documentation and communication with LLMs4.5.1 Generating code comments4.5.2 Generating release notes4.6 Maintaining a balance with code assistantsSummary
5 Test planning with AI support
5.1 Defining test planning in modern testing5.1.1 Test planning, LLMs, and area of effect5.2 Focused prompts with the use of models5.2.1 Weak prompts mean weak suggestions5.2.2 What are models and why can they help5.3 Combining models and LLMs to assist test planning5.3.1 Creating a model to identify prompts5.3.2 Experimenting with different model types5.4 LLMs and test cases5.4.1 Having a healthy skepticism of generated risks and test casesSummary
6 Rapid data creation using AI
6.1 Generating and transforming data with LLMs6.1.1 Prompting LLMs to generate simple data sets6.1.2 Transforming test data into different formats6.2 Processing complex test data with LLMs6.2.1 Using format standards in prompts6.2.2 SQL exports as prompt guides6.3 Setting up LLMs as test data managers6.3.1 Setting up an OpenAI account6.3.2 Connecting to OpenAI6.4 Benefiting from generated test dataSummary
7 Accelerating and improving UI automation using AI
7.1 Rapidly creating UI automation7.1.1 Setting up a project7.1.2 Creating our initial check with ChatGPT support7.1.3 Filling in gaps from generated code7.2 Improving existing UI automation7.2.1 Updating state management to use an appropriate layer7.2.2 Getting into the groove with AI toolsSummary
8 Assisting exploratory testing with artificial intelligence
8.1 Organizing exploratory testing with LLMs8.1.1 Augmenting identified risks with LLMs8.1.2 Augmenting charter lists with LLMs8.2 Using LLMs during exploratory testing8.2.1 Establishing an understanding8.2.2 Creating data requirements for a session8.2.3 Exploring and investigating bugs8.2.4 Using LLMs to assist exploratory testing8.3 Summarizing testing notes with LLMsSummary
9 AI agents as testing assistants
9.1 Understanding AI agents and LLMs9.1.1 What defines an AI agent?9.1.2 How an agent works with LLMs9.2 Creating an AI Test Assistant9.2.1 Setting up our dummy AI agent9.2.2 Giving our AI agent functions to execute9.2.3 Chaining tools together9.3 Moving forward with AI test assistants9.3.1 Examples of AI test assistants9.3.2 Handling the challenges of working with agentsSummary
Part 3. Context: Customizing LLMs for testing contexts
10 Introducing customized LLMs
10.1 The challenge with LLMs and context10.1.1 Tokens, context windows, and limitations10.1.2 Embedding context as a solution10.2 Embedding context further into prompts and LLMs10.2.1 RAG10.2.2 Fine-tuning LLMs10.2.3 Comparing the two approaches10.2.4 Combining RAG and fine-tuningSummary
11 Contextualizing prompts with retrieval-augmented generation
11.1 Extending prompts with RAG11.2 Building a RAG setup11.2.1 Building our RAG framework11.2.2 Testing our RAG framework11.3 Enhancing data storage for RAG11.3.1 Working with Vector databases11.3.2 Setting up a vector-database-backed RAG11.3.3 Testing a Vector-database-backed RAG framework11.3.4 Going forward with RAG frameworksSummary
12 Fine-tuning LLMs with business domain knowledge
12.1 Exploring the fine-tuning process12.1.1 A map of the fine-tuning process12.1.2 Goal setting12.2 Executing a fine-tuning session12.2.1 Preparing data for training12.2.2 Preprocessing and setup12.2.3 Working with fine-tuning tools12.2.4 Setting off a fine-tuning run12.2.5 Testing the results of a fine-tune12.2.6 Lessons learned with fine-tuningSummary
Appendix A. Setting up and using ChatGPT
Appendix B. Setting up and using GitHub Copilot
B.1 Setting up CopilotB.1.1 Setting up a Copilot accountB.1.2 Installing the Copilot pluginB.1.3 Granting access to your Copilot accountB.2 Working with CopilotB.2.1 Exploring suggestions
Appendix C. Exploratory testing notes
index

Overview

Speed up your testing and deliver exceptional product quality with the power of AI tools.

The more you test, the more you learn about your software. Software Testing with Generative AI shows you how you can expand, automate, and enhance your testing with Large Language Model (LLM)-based AI. Your team will soon be delivering higher quality tests, all in less time.

In Software Testing with Generative AI you’ll learn how to:

Spot opportunities to improve test quality with AI
Construct test automation with the support of AI tools
Formulate new ideas during exploratory testing using AI tools
Use AI tools to aid the design process of new features
Improve the testability of a context with the help of AI tools
Maximize your output with prompt engineering
Create custom LLMs for your business’s specific needs

Software Testing with Generative AI is full of hype-free advice for supporting your software testing with AI. In it, you’ll find strategies from bestselling author Mark Winteringham to generate synthetic testing data, implement automation, and even augment and improve your test design with AI.

About the Technology
There’s a simple rule in software testing: the more you test, the more you learn. And as any testing pro will tell you, good testing takes time. By integrating large language models (LLMs) and generative AI into your process, you can dramatically automate and enhance testing, improve quality and coverage, and deliver more meaningful results.

About the Book
Software Testing with Generative AI shows you how AI can elevate every aspect of testing—automation, test data management, test scripting, exploratory testing, and more! Learn how to use AI coding tools like Copilot to guide test-driven development, get relevant feedback about your applications from ChatGPT, and use the OpenAI API to integrate AI into your data generation. You’ll soon have higher-quality testing that takes up less of your time.

What's Inside

Improve test quality and coverage
AI-powered test automation
Build agents that act as testing assistants

About the Reader
For developers, testers, and quality engineers.

About the Author
Mark Winteringham is an experienced software tester who teaches many aspects of software testing. He is the author of Testing Web APIs.

The technical editor on this book was Robert Walsh.

Quotes
Pragmatic approach and real-world examples show you how to deliver higher quality software in less time.
- Ankit Virmani, Data and ML Architect, Google

Full of ideas on how to streamline software testing processes by using GenAI.
- Dr. Ron Hübler, Software Engineer Lead, Capgemini

The real-world examples demonstrate the potential of integrating Generative AI into software testing.
- Laurence Giglio, QA Consultant, FactFindersPro.com

Great to see testing techniques condensed into prompts that are usable for everyone!
- Marvin Schwarze, Data Engineer, REWE Group

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781633437364Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills