Quantifying the User Experience

Chapter 5

Is There a Statistical Difference between Designs?

Introduction

Many researchers first realize the need for statistics when they have to compare two designs or products in an A/B test or competitive analysis. When stakes are high (or subject to scrutiny), just providing descriptive statistics and declaring one design better is insufficient. What is needed is to determine whether the difference between designs (e.g., among conversion rates, task times, or ratings) is greater than what we’d expect from chance. This chapter is all about determining whether a difference is statistically significant and how large or small of a difference likely exists in the untested population.

Comparing Two Means (Rating Scales and Task Times)

A central ...