UX for Beginners

Chapter 24. How Many Users Do You Need?

Should you go for the wisdom of the crowd, or should you trust your most loyal users? Should you ask the people who know the most about your designs, or the beginners who have fresh eyes?

This is a very common question, and a good one. How many users should you include in your user research to make sure that you get all the information you need? Well, it depends, actually.

The Less Obvious the Problem, the More People You Need to Find It

Let’s say there are two problems with your design:

1. Most people don’t notice the button that opens the menu.

2. Your pricing page makes it seem like your product is not free, but it is.

Both are real issues I have seen in testing.

Let’s say the menu problem affects one out of every three people. That means you need at least three users to find the problem with the menu (it’s more like four or five in real life).

And let’s say the pricing problem affects 1 out of every 20 people. That means you need at least 20 users to find the problem with the menu (again, it’s more like 30 or 40 in real life).

So if you get five test users—a common practice for user testing—you’ll probably find the menu problem and miss the pricing problem. D’oh!

Since real services often only convert a few people out of 20 anyway, that pricing issue might be a significant amount of your sales! That is why face-to-face testing is useful, but not reliable in isolation.

Get Users Who Fit the Profile and A Couple Who Don’t

In Chapter 29, Creating User Profiles ...