book

Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests, Second Edition

Name: Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests, Second Edition
ISBN: 9780470185483

by Jared Spool, Dana Chisnell, Jeffrey Rubin

May 2008

Intermediate to advanced

384 pages

9h 53m

English

Wiley

Read now

Unlock full access

Copyright
About the Authors
Credits
Acknowledgments
Foreword
Preface to the Second Edition
1. Usability Testing: An Overview
1. What Makes Something Usable?
1.1. What Do We Mean by "Usable"?1.2. What Makes Something Less Usable?1.2.1. Five Reasons Why Products Are Hard to Use1.2.1.1. Reason 1: Development Focuses on the Machine or System1.2.1.2. Reason 2: Target Audiences Expand and Adapt1.2.1.3. Reason 3: Designing Usable Products Is Difficult1.2.1.4. Reason 4: Team Specialists Don't Always Work in Integrated Ways1.2.1.5. Reason 5: Design and Implementation Don't Always Match1.3. What Makes Products More Usable?1.3.1.1.3.1.1. An Early Focus on Users and Tasks1.3.1.2. Evaluation and Measurement of Product Usage1.3.1.3. Iterative Design and Testing1.3.2. Attributes of Organizations That Practice UCD1.3.2.1. Phases That Include User Input1.3.2.2. A Multidisciplinary Team Approach1.3.2.3. Concerned, Enlightened Management1.3.2.4. A "Learn as You Go" Perspective1.3.2.5. Defined Usability Goals and Objectives1.4. What Are Techniques for Building in Usability?1.4.1. Ethnographic Research1.4.2. Participatory Design1.4.3. Focus Group Research1.4.4. Surveys1.4.5. Walk-Throughs1.4.6. Open and Closed Card Sorting1.4.7. Paper Prototyping1.4.8. Expert or Heuristic Evaluations1.4.9. Usability Testing1.4.10. Follow-Up Studies
2. What Is Usability Testing?
2.1. Why Test? Goals of Testing2.1.1. Informing Design2.1.2. Eliminating Design Problems and Frustration2.1.3. Improving Profitability2.2. Basics of the Methodology2.2.1. Basic Elements of Usability Testing2.2.2. Limitations of Testing
3. When Should You Test?
3.1. Our Types of Tests: An Overview3.2. Exploratory or Formative Study3.2.1. When3.2.2. Objective3.2.3. Overview of the Methodology3.2.4. Example of Exploratory Study3.3. Assessment or Summative Test3.3.1. When3.3.2. Objective3.3.3. Overview of the Methodology3.4. Validation or Verification Test3.4.1. When3.4.2. Objective3.4.3. Overview of the Methodology3.5. Comparison Test3.5.1. When3.5.2. Objective3.5.3. Overview of the Methodology3.6. Iterative Testing: Test Types through the Lifecycle3.6.1. Test 1: Exploratory/Comparison Test3.6.1.1. The situation3.6.1.2. Main Research Questions3.6.1.3. Brief Summary of Outcome3.6.2. Test 2: Assessment Test3.6.2.1. The Situation3.6.2.2. Main Test Objectives3.6.2.3. Brief Summary of Test Outcome3.6.3. Test 3: Verification Test3.6.3.1. The Situation3.6.3.2. Test Objectives3.6.3.3. Brief Summary of Test Outcome

4. Skills for Test Moderators
4.1.4.1.1. Who Should Moderate?4.1.2. Human Factors Specialist4.1.3. Marketing Specialist4.1.4. Technical Communicator4.1.5. Rotating Team Members4.1.6. External Consultant4.2. Characteristics of a Good Test Moderator4.2.1. Grounding in the Basics of User-Centered Design4.2.2. Quick Learner4.2.3. Instant Rapport with Participants4.2.4. Excellent Memory4.2.5. Good Listener4.2.6. Comfortable with Ambiguity4.2.7. Flexibility4.2.8. Long Attention Span4.2.9. Empathic "People Person"4.2.10. "Big Picture" Thinker4.2.11. Good Communicator4.2.12. Good Organizer and Coordinator4.3. Getting the Most out of Your Participants4.3.1. Choose the Right Format4.3.1.1. Sit-By Sessions versus Observing from Elsewhere4.3.1.2. "Think-Aloud" Advantages and Disadvantages4.3.1.3. Retrospective Review4.3.2. Give Participants Time to Work through Hindrances4.3.3. Offer Appropriate Encouragement4.4. Troubleshooting Typical Moderating Problems4.4.1. Leading Rather Than Enabling4.4.2. Too Involved with the Act of Data Collection4.4.3. Acting Too Knowledgeable4.4.4. Too Rigid with the Test Plan4.4.5. Not Relating Well to Each Participant4.4.6. Jumping to Conclusions4.5. How to Improve Your Session-Moderating Skills4.5.1. Learn the Basic Principles of Human Factors/Ergonomics4.5.2. Learn from Watching Others4.5.3. Watch Yourself on Tape4.5.4. Work with a Mentor4.5.5. Practice Moderating Sessions4.5.6. Learn to Meditate4.5.7. Practice "Bare Attention"
2. The Process for Conducting a Test
5. Develop the Test Plan
5.1. Why Create a Test Plan?5.1.1. It Serves as a Blueprint for the Test5.1.2. It Serves as the Main Communication Vehicle5.1.3. It Defines or Implies Required Resources5.1.4. It Provides a Focal Point for the Test and a Milestone5.2. The Parts of a Test Plan5.2.1. Review the Purpose and Goals of the Test5.2.1.1. When Not to Test5.2.1.2. Good Reasons to Test5.2.2. Communicate Research Questions5.2.3. Summarize Participant Characteristics5.2.4. Describe the Method5.2.4.1. Independent Groups Design or Between Subjects Design5.2.4.2. Within-Subjects Design5.2.4.3. Testing Multiple Product Versions5.2.4.4. Testing Multiple User Groups5.2.5. List the Tasks5.2.5.1. Parts of a Task for the Test Plan5.2.5.2. Tips for Developing the Task List5.2.5.3. Example Task: Navigation Tab on a Web Site5.2.5.4. Ways to Prioritize Tasks5.2.6. Describe the Test Environment, Equipment, and Logistics5.2.7. Explain What the Moderator Will Do5.2.8. List the Data You Will Collect5.2.8.1. Sample Performance Measures5.2.8.2. Qualitative Data5.2.8.3. Sample Preference Measures5.2.9. Describe How the Results Will Be Reported5.3. Sample Test Plan
6. Set Up a Testing Environment
6.1. Decide on a Location and Space6.1.1. In a Lab or at the User's Site?6.1.2. Test in Multiple Geographic Locations?6.1.3. Arranging Sessions at a User's Site6.1.3.1. Minimalist Portable Test Lab6.1.4. Setting up a Permanent or Fixed Test Lab6.1.4.1. Simple Single-Room Setup6.1.4.2. Modified Single-Room Setup6.1.4.3. Large Single-Room Setup6.1.4.4. Electronic Observation Room Setup6.1.4.5. Classic Testing Laboratory Setup6.2. Recommended Testing Environment: Minimalist Portable Lab6.3. Gather and Check Equipment, Artifacts, and Tools6.3.1. Basic Equipment, Tools, and Props6.3.2. Gathering Biometric Data6.4. Identify Co-Researchers, Assistants, and Observers6.4.1. Data Gatherer/Note Taker6.4.2. Timekeeper6.4.3. Product/Technical Expert(s)6.4.4. Additional Testing Roles6.4.5. Test Observers
7. Find and Select Participants
7.1. Characterize Users7.1.1. Visualize the Test Participant7.1.2. Differentiate between Purchaser and End User7.1.3. Look for Information about Users7.1.3.1. Requirements and Specification Documents7.1.3.2. Structured Analyses or Marketing Studies7.1.3.3. Product Manager (R&D)7.1.3.4. Product Manager (Marketing)7.1.3.5. Competitive Benchmarking and Analysis Group7.2. Define the Criteria for Each User Group7.2.1. Define Expertise7.2.2. Specify Requirements and Classifiers for Selection7.2.3. Document the User Profile7.2.4. Divide the User Profile into Distinct Categories7.2.5. Consider a Matrix Test Design7.3. Determine the Number of Participants to Test7.4. Write the Screening Questionnaire7.4.1. Review the Profile to Understand Users' Backgrounds7.4.2. Identify Specific Selection Criteria7.4.3. Formulate Screening Questions7.4.4. Organize the Questions in a Specific Order7.4.5. Develop a Format for Easy Flow through the Questionnaire7.4.6. Test the Questionnaire on Colleagues and Revise It7.4.7. Consider Creating an "Answer Sheet"7.5. Find Sources of Participants7.5.1. Internal Participants7.5.2. Qualified Friends and Family7.5.3. Web Site Sign-Up7.5.4. Existing Customers from In-House Lists7.5.5. Existing Customers through Sales Representatives7.5.6. User Groups or Clubs, Churches, or Other Community Groups7.5.7. Societies and Associations7.5.8. Referrals from Personal Networks, Coworkers, and Other Participants7.5.9. Craigslist7.5.10. College Campuses7.5.11. Market Research Firms or Recruiting Specialists7.5.12. Employment Agencies7.5.13. Newspaper Advertisements7.6. Screen and Select Participants7.6.1. Screening Considerations7.6.1.1. Use the Questionnaire or Open-Ended Interview Questions?7.6.1.2. Complete the Screener Always, or Only When Fully Qualified?7.6.2. Conduct Screening Interviews7.6.2.1. Inform the Potential Participant Who You Are7.6.2.2. Explain Why You are Calling and How You Got the Contact Information7.6.2.3. Go through the Questions in the Questionnaire7.6.2.4. As You Eliminate or Accept People, Mark Them Off on Your List7.6.3. Include a Few Least Competent Users in Every Testing Sample7.6.4. Beware of Inadvertently Testing Only the "Best" People7.6.5. Expect to Make Tradeoffs7.7. Schedule and Confirm Participants7.7.1. Compensate Participants7.7.2. Protect Participants' Privacy and Personal Information
8. Prepare Test Materials
8.1. Guidelines for Observers8.2. Orientation Script8.2.1. Keep the Tone of the Script Professional, but Friendly8.2.2. Keep the Speech Short8.2.3. Plan to Read the Script to Each Participant Verbatim8.2.4. Write the Orientation Script Out8.2.4.1. Make Introductions8.2.4.2. Offer Refreshments8.2.4.3. Explain Why the Participant Is Here8.2.4.4. Describe the Testing Setup8.2.4.5. Explain What Is Expected of the Participant8.2.4.6. Assure the Participant That He or She Is Not Being Tested8.2.4.7. Explain Any Unusual Requirements8.2.4.8. Mention That It Is Okay to Ask Questions at Any Time8.2.4.9. Ask for Any Questions8.2.4.10. Refer to Any Forms That Need to Be Completed and Pass Them Out8.3. Background Questionnaire8.3.1. Focus on Characteristics That May Influence Performance8.3.2. Make the Questionnaire Easy to Fill Out and Compile8.3.3. Test the Questionnaire8.3.4. Decide How to Administer the Questionnaire8.4. Data Collection Tools8.4.1. Review the Research Question(s) Outlined in Your Test Plan8.4.2. Decide What Type of Information to Collect8.4.3. Select a Data Collection Method8.4.3.1. Fully Automated Data Loggers8.4.3.2. Online Data Collection8.4.3.3. User-Generated Data Collection8.4.3.4. Manual Data Collection8.4.3.5. Other Data Collection Methods8.5. Nondisclosures, Consent Forms, and Recording Waivers8.6. Pre-Test Questionnaires and Interviews8.6.1. Discover Attitudes and First Impressions8.6.2. Learn about Whether Participants Value the Product8.6.3. Qualify Participants for Inclusion into One Test Group or Another8.6.4. Establish the Participant's Prerequisite Knowledge Prior to Using the Product8.7. Prototypes or Products to Test8.8. Task Scenarios8.8.1. Provide Realistic Scenarios, Complete with Motivations to Perform8.8.2. Sequence the Task Scenarios in Order8.8.3. Match the Task Scenarios to the Experience of the Participants8.8.4. Avoid Using Jargon and Cues8.8.5. Try to Provide a Substantial Amount of Work in Each Scenario8.8.6. Give Participants the Tasks to Do8.8.6.1. Reading Task Scenarios to the Participants8.8.6.2. Letting the Participants Read Task Scenarios Themselves8.9. Optional Training Materials8.9.1. Ensure Minimum Expertise8.9.2. Get a View of the User after Experiencing the Product8.9.3. You Want to Test Features for Advanced Users8.9.4. What Are the Benefits of Prerequisite Training?8.9.4.1. You Can Conduct a More Comprehensive, Challenging Usability Test8.9.4.2. You Can Test Functionality That Might Otherwise Get Overlooked During a Test8.9.4.3. Developing the Training Forces You to Understand How Someone Learns to Use Your Product8.9.5. Some Common Questions about Prerequisite Training8.10. Post-Test Questionnaire8.10.1. Use the Research Questions(s) from the Test Plan as the Basis for Your Content8.10.2. Develop Questionnaires That Will Be Distributed Either during or after a Session8.10.3. Ask Questions Related to That Which You Cannot Directly Observe8.10.4. Develop the Basic Areas and Topics You Want to Cover8.10.5. Design the Questions and Responses for Simplicity and Brevity8.10.6. Use the Pilot Test to Refine the Questionnaire8.11. Common Question Formats8.11.1. Likert Scales8.11.2. Semantic Differentials8.11.3. Fill-In Questions8.11.4. Checkbox Questions8.11.5. Branching Questions8.12. Debriefing Guide
9. Conduct the Test Sessions
9.1. Guidelines for Moderating Test Sessions9.1.1. Moderate the Session Impartially9.1.2. Be Aware of the Effects of Your Voice and Body Language9.1.3. Treat Each New Participant as an Individual9.1.4. If Appropriate, Use the "Thinking Aloud" Technique9.1.4.1. Advantages of the "Thinking Aloud" Technique9.1.4.2. Disadvantages of the "Thinking Aloud" Technique9.1.4.3. How to Enhance the "Thinking Aloud" Technique9.1.5. Probe and Interact with the Participant as Appropriate9.1.6. Stay Objective, But Keep the Tone Relaxed9.1.7. Don't "Rescue" Participants When They Struggle9.1.8. If You Make a Mistake, Continue On9.1.9. Ensure That Participants Are Finished Before Going On9.1.10. Assist the Participants Only as a Last Resort9.1.10.1. When to Assist9.1.10.2. How to Assist9.2. Checklists for Getting Ready9.2.1. Checklist 1: A Week or So before the Test9.2.1.1. Take the Test Yourself9.2.1.2. Conduct a Pilot Test9.2.1.3. Revise the Product9.2.1.4. Check Out All the Equipment and the Testing Environment9.2.1.5. Request a Temporary "Freeze" on Development9.2.2. Checklist 2: One Day before the Test9.2.2.1. Check That the Video Equipment Is Set Up and Ready9.2.2.2. Check That the Product, if Software or Hardware, Is Working9.2.2.3. Assemble All Written Test Materials9.2.2.4. Check on the Status of Your Participants9.2.2.5. Double-Check the Test Environment and Equipment9.2.3. Checklist 3: The Day of the Test9.2.3.1. Prepare Yourself Mentally9.2.3.2. Greet the Participant9.2.3.3. Have the Participant Fill Out and Sign Any Preliminary Documents9.2.3.4. Read the Orientation Script and Set the Stage9.2.3.5. Have the Participant Fill Out Any Pre-Test Questionnaires9.2.3.6. Move to the Testing Area and Prepare to Test9.2.3.7. Start Recordings9.2.3.8. Set Decorum for Observers in the Room9.2.3.9. Provide Any Prerequisite Training If Your Test Plan Includes It9.2.3.10. Either Distribute or Read the Written Task Scenario(s) to the Participant9.2.3.11. Record Start Time, Observe the Participant, and Collect All Critical Data9.2.3.12. Have the Participant Complete All Post-Test Questionnaires9.2.3.13. Debrief the Participant9.2.3.14. Close the Session9.2.3.15. Organize Data Collection and Observation Sheets9.2.3.16. Debrief with Observers9.2.3.17. Provide Adequate Time between Test Sessions9.2.3.18. Prepare for the Next Participant9.3. When to Intervene9.3.1. When to Deviate from the Test Plan9.4. What Not to Say to Participants
10. Debrief the Participant and Observers
10.1. Why Review with Participants and Observers?10.2. Techniques for Reviewing with Participants10.3. Where to Hold the Participant Debriefing Session10.4. Basic Debriefing Guidelines10.5. Advanced Debriefing Guidelines and Techniques10.5.1. "Replay the Test" Technique10.5.1.1. The Manual Method10.5.1.2. The Video Method10.5.2. Audio Record the Debriefing Session10.5.3. Reviewing Alternate Designs10.5.4. "What Did You Remember?" Technique10.5.5. "Devil's Advocate" Technique10.5.5.1. How to Implement the "Devil's Advocate" Technique10.5.5.2. Example of the "Devil's Advocate" Technique10.6. Reviewing and Reaching Consensus with Observers10.6.1. Why Review with Observers?10.6.2. Between Sessions10.6.3. At the End of the Study
11. Analyze Data and Observations
11.1. Compile Data11.1.1. Begin Compiling Data as You Test11.1.2. Organize Raw Data11.2. Summarize Data11.2.1. Summarize Performance Data11.2.1.1. Task Accuracy11.2.1.2. Task Timings11.2.2. Summarize Preference Data11.2.3. Compile and Summarize Other Measures11.2.4. Summarize Scores by Group or Version11.3. Analyze Data11.3.1. Identify Tasks That Did Not Meet the Success Criterion11.3.2. Identify User Errors and Difficulties11.3.3. Conduct a Source of Error Analysis11.3.4. Prioritize Problems11.3.5. Analyze Differences between Groups or Product Versions11.3.6. Using Inferential Statistics
12. Report Findings and Recommendations
12.1. What Is a Finding?12.2. Shape the Findings12.3. Draft the Report12.3.1. Why Write a Report?12.3.2. Organize the Report12.3.2.1. Executive Summary12.3.2.2. Method12.3.2.3. Results12.3.2.4. Findings and Recommendations (Discussion)12.4. Develop Recommendations12.4.1. Focus on Solutions That Will Have the Widest Impact12.4.2. Ignore Political Considerations for the First Draft12.4.3. Provide Both Short-Term and Long-Term Recommendations12.4.4. Indicate Areas Where Further Research Is Required12.4.5. Be Thorough12.4.6. Make Supporting Material Available to Reviewers12.5. Refine the Report Format12.6. Create a Highlights Video or Presentation12.6.1. Cautions about Highlights12.6.2. Steps for Producing a Highlights Video12.6.2.1. Consider the Points You Want to Make12.6.2.2. Set Up a Spreadsheet to Plan and Document the Video12.6.2.3. Pick the Clips12.6.2.4. Review Timing and Organization12.6.2.5. Draft Titles and Captions12.6.2.6. Review and Wrap
3. Advanced Techniques
13. Variations on the Basic Method
13.1. Who? Testing with Special Populations13.1.1. People Who Have Disabilities13.1.1.1. Scheduling and Reminding13.1.1.2. During the Session13.1.2. Older Adults13.1.2.1. Scheduling and Reminding13.1.2.2. During the Session13.1.3. Children13.1.3.1. Scheduling and Reminding13.1.3.2. During the Session13.2. What: Prototypes versus Real Products13.2.1.13.2.1.1. Paper and Other Low-Fi Prototypes13.2.1.2. Clickable or Usable Prototypes13.3. How? Techniques for Monitored Tests13.3.1. Flexible Scripting13.3.1.1. What You Get13.3.1.2. How to Use It13.3.2. Gradual Disclosure or Graduated Prompting13.3.2.1. What You Get13.3.2.2. How to Use It13.3.3. Co-Discovery (Two Participants at a Time)13.3.3.1. What You Get13.3.3.2. How to Use It13.3.4. Alpha or Beta Testing with Favored Clients13.3.4.1. What You Get13.3.4.2. How to Use It13.3.5. Play Tests13.3.5.1. What You Get13.3.5.2. How to Use It13.4. Where? Testing Outside a Lab13.4.1. Remote Testing13.4.1.1. What You Get13.4.1.2. How to Use It13.4.2. Automated Testing13.4.2.1. What You Get13.4.2.2. How to Use It13.4.3. Testing In-Home or On-Site13.4.3.1. What You Get13.4.3.2. How to Use It13.5. Self-Reporting (Surveys, Diary Studies)13.5.1.13.5.1.1. What You Get13.5.1.2. How to Use It
14. Expanding from Usability Testing to Designing the User Experience
14.1. Stealth Mode: Establish Value14.1.1. Choose the First Project Carefully14.1.1.1. Begin Your Education14.1.1.2. Start Slowly and Conservatively, Get Buy-In14.1.1.3. Volunteer Your Services14.1.1.4. Create a Strategy and Business Case14.2. Build on Successes14.2.1. Set Up Long-Term Relationships14.2.2. Sell Yourself and What You Are Doing14.2.3. Strategize: Choose Your Battles Carefully14.3. Formalize Processes and Practices14.3.1. Establish a Central Residency for User-Centered Design14.3.2. Add Usability-Related Activities to the Product Life Cycle14.3.3. Educate Others within Your Organization14.3.4. Identify and Cultivate Champions14.3.5. Publicize the Usability Success Stories14.3.6. Link Usability to Economic Benefits14.4. Expand UCD throughout the Organization14.4.1. Pursue More Formal Educational Opportunities14.4.2. Standardize Participant Recruitment Policies and Procedures14.4.3. Align Closely with Market Research and Industrial Design14.4.4. Evaluate Product Usability in the Field after Product Release14.4.5. Evaluate the Value of Your Usability Engineering Efforts14.4.6. Develop Design Standards14.4.7. Focus Your Efforts Early in the Product Life Cycle14.4.8. Create User Profiles, Personas, and Scenarios
A. Afterword

Content preview from Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests, Second Edition

Chapter 13. Variations on the Basic Method

Having previously described the basic method for conducting a usability test, with one participant and one test moderator, this chapter describes some variations of that technique. It discusses some common variations, with tips about implementing them, such as testing with special populations, working with prototypes, and techniques for conducting sessions inside and outside a lab, including remote and automated testing. Ultimately, your testing methodology is limited only by your imagination and the types of questions you need to have answered in testing. We include many references to books and articles that expand on the techniques in this chapter on the web site that accompanies this book: www.wiley.com/go/usabilitytesting.

Who? Testing with Special Populations

Chapter 7 is all about recruiting participants, but following are some special considerations for people who you may want or need to include in your usability test.

People Who Have Disabilities

You probably have sought out this section of the book because your company wants to meet U.S. federal requirements for accessibility, such as the Americans with Disabilities Act (ADA 508). Or perhaps your company is particularly enlightened and understands that there is a huge potential target market in which there are people with disabilities who use technology regularly. If either of these cases rings a bell, you should be including people with disabilities in your usability tests. In our ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Business Model Generation: A Handbook for Visionaries, Game Changers, and Challengers

Publisher Resources

ISBN: 9780470185483Purchase book

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design