book

Software Testing with Generative AI

Name: Software Testing with Generative AI
Author: Mark Winteringham
ISBN: 9781633437364

by Mark Winteringham

December 2024

Intermediate to advanced

304 pages

8h 57m

English

Manning Publications

Read now

Unlock full access

Software Testing with Generative AI
Copyright
dedication
contents
front matter
forewordprefaceacknowledgmentsabout this bookWho should read this bookHow this book is organized: A road mapAbout the codeliveBook discussion forumabout the authorabout the cover illustration
Part 1. Mindset: Establishing a positive relationship with LLMs
1 Enhancing testing with large language models
1.1 Recognizing the effect of AI tools on testing and development1.1.1 Data generation1.1.2 Automated test building1.1.3 Test design1.2 Delivering value with LLMs1.2.1 A model for delivering value1.2.2 Using both human and AI abilities1.2.3 Being skeptical of LLMsSummary
2 Large language models and prompt engineering
2.1 LLMs explained2.2 Avoiding the risks of using LLMs2.2.1 Hallucinations2.2.2 Data provenance2.2.3 Data privacy2.3 Improving results with prompt engineering2.4 Examining the principles of prompt engineering2.4.1 Principle 1: Write clear and specific instructions2.4.2 Tactic 1: Use delimiters2.4.3 Tactic 2: Ask for structured output2.4.4 Tactic 3: Check for assumptions2.4.5 Tactic 4: Few-shot prompting2.4.6 Principle 2: Give the model time to “think”2.4.7 Tactic 1: Specify the steps to complete the task2.4.8 Tactic 2: Instruct the model to work out its own solution first2.5 Working with various LLMs2.5.1 Comparing LLMs2.5.2 Examining popular LLMs2.6 Creating a library of prompts2.7 Solving problems by using promptsSummary
3 Artificial intelligence, automation, and testing
3.1 The value of testing3.1.1 A different way of thinking about testing3.1.2 A holistic approach to testing3.2 How tools help with testing3.2.1 Automation bias3.2.2 Being selective with tooling3.3 Knowing when to use LLMs in testing3.3.1 Generative capabilities3.3.2 Transformation capabilities3.3.3 Enhancing capabilities3.3.4 LLMs in use in testingSummary
Part 2. Technique: Task identification and prompt engineering in testing

4 AI-assisted testing for developers
4.1 Examining the rise of the automated developer4.2 Pairing with LLMs4.2.1 Analyzing ideas4.2.2 Analyzing code4.2.3 Recognizing that a simulation is better than nothing at all4.3 Building in quality with AI assistance4.4 Creating our first TDD loop with LLMs4.4.1 Preparing the work4.4.2 Loop 1: Save a timesheet entry4.4.3 Loop 2: Retrieve a timesheet entry4.4.4 Loop 3: Calculating times for a project4.4.5 Refactoring code4.5 Improving documentation and communication with LLMs4.5.1 Generating code comments4.5.2 Generating release notes4.6 Maintaining a balance with code assistantsSummary
5 Test planning with AI support
5.1 Defining test planning in modern testing5.1.1 Test planning, LLMs, and area of effect5.2 Focused prompts with the use of models5.2.1 Weak prompts mean weak suggestions5.2.2 What are models and why can they help5.3 Combining models and LLMs to assist test planning5.3.1 Creating a model to identify prompts5.3.2 Experimenting with different model types5.4 LLMs and test cases5.4.1 Having a healthy skepticism of generated risks and test casesSummary
6 Rapid data creation using AI
6.1 Generating and transforming data with LLMs6.1.1 Prompting LLMs to generate simple data sets6.1.2 Transforming test data into different formats6.2 Processing complex test data with LLMs6.2.1 Using format standards in prompts6.2.2 SQL exports as prompt guides6.3 Setting up LLMs as test data managers6.3.1 Setting up an OpenAI account6.3.2 Connecting to OpenAI6.4 Benefiting from generated test dataSummary
7 Accelerating and improving UI automation using AI
7.1 Rapidly creating UI automation7.1.1 Setting up a project7.1.2 Creating our initial check with ChatGPT support7.1.3 Filling in gaps from generated code7.2 Improving existing UI automation7.2.1 Updating state management to use an appropriate layer7.2.2 Getting into the groove with AI toolsSummary
8 Assisting exploratory testing with artificial intelligence
8.1 Organizing exploratory testing with LLMs8.1.1 Augmenting identified risks with LLMs8.1.2 Augmenting charter lists with LLMs8.2 Using LLMs during exploratory testing8.2.1 Establishing an understanding8.2.2 Creating data requirements for a session8.2.3 Exploring and investigating bugs8.2.4 Using LLMs to assist exploratory testing8.3 Summarizing testing notes with LLMsSummary
9 AI agents as testing assistants
9.1 Understanding AI agents and LLMs9.1.1 What defines an AI agent?9.1.2 How an agent works with LLMs9.2 Creating an AI Test Assistant9.2.1 Setting up our dummy AI agent9.2.2 Giving our AI agent functions to execute9.2.3 Chaining tools together9.3 Moving forward with AI test assistants9.3.1 Examples of AI test assistants9.3.2 Handling the challenges of working with agentsSummary
Part 3. Context: Customizing LLMs for testing contexts
10 Introducing customized LLMs
10.1 The challenge with LLMs and context10.1.1 Tokens, context windows, and limitations10.1.2 Embedding context as a solution10.2 Embedding context further into prompts and LLMs10.2.1 RAG10.2.2 Fine-tuning LLMs10.2.3 Comparing the two approaches10.2.4 Combining RAG and fine-tuningSummary
11 Contextualizing prompts with retrieval-augmented generation
11.1 Extending prompts with RAG11.2 Building a RAG setup11.2.1 Building our RAG framework11.2.2 Testing our RAG framework11.3 Enhancing data storage for RAG11.3.1 Working with Vector databases11.3.2 Setting up a vector-database-backed RAG11.3.3 Testing a Vector-database-backed RAG framework11.3.4 Going forward with RAG frameworksSummary
12 Fine-tuning LLMs with business domain knowledge
12.1 Exploring the fine-tuning process12.1.1 A map of the fine-tuning process12.1.2 Goal setting12.2 Executing a fine-tuning session12.2.1 Preparing data for training12.2.2 Preprocessing and setup12.2.3 Working with fine-tuning tools12.2.4 Setting off a fine-tuning run12.2.5 Testing the results of a fine-tune12.2.6 Lessons learned with fine-tuningSummary
Appendix A. Setting up and using ChatGPT
Appendix B. Setting up and using GitHub Copilot
B.1 Setting up CopilotB.1.1 Setting up a Copilot accountB.1.2 Installing the Copilot pluginB.1.3 Granting access to your Copilot accountB.2 Working with CopilotB.2.1 Exploring suggestions
Appendix C. Exploratory testing notes
index

Content preview from Software Testing with Generative AI

10 Introducing customized LLMs

This chapter covers

How a lack of context affects an LLM’s performance
How RAG works and its value
How the fine-tuning of LLMs works and its value
Comparing RAG and fine-tuning approaches

Over the past few chapters, we saw how to hone our skills to identify distinct, focused tasks that large language models (LLMs) can support. Combined with a range of prompt-engineering techniques, we’ve been successful in getting LLMs to return responses that are valuable for our testing activities. However, despite the lessons we learned, the responses we receive might still not be completely aligned with our needs and context. Although it would be foolish to think that we can completely align an LLM with our context, there ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Generative AI for Software Development

Sergio Pereira

Generative AI with LangChain

Ben Auffarth

Prompt Engineering for Generative AI

James Phoenix, Mike Taylor

AI Codecon: Coding with AI—The End of Software Development as We Know It

Tim O'Reilly, Addy Osmani, Gergely Orosz, Kent Beck, Camille Fournier, Avi Flombaum, Maxi Ferreira, Harper Reed, Jay Parikh, Birgitta Böckeler, Angie Jones, Craig McLuckie, Patty O’Callaghan, Chip Huyen, swyx, Andrew Stellman, Phillip Carter, Nikola Balic, Brett Smith, Chelsea Troy, Lili Jiang, Iyanuoluwa Ajao

Publisher Resources

ISBN: 9781633437364Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Software Testing with Generative AI

by Mark Winteringham

10 Introducing customized LLMs

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.