Complete Web Monitoring

Deciding Who to Ask

How many people do you want to survey? The simple answer is, of course, as many as possible. One commonly cited reason for surveying many people is to ensure that data is statistically significant—the more respondents, the more confidence you have in your results.

Remember, however, that your survey is already biased based on the people who responded to it. Much of what you collect won’t be applicable to your site’s visitors as a whole, anyway. You still need a large sample, but not because it will accurately model your entire market. The real reasons for wanting many responses are twofold: to validate patterns, and to segment.

You need to know which responses aren’t complete outliers, and for this, you need dozens, even hundreds, of answers. Many of the clustering and visualization tools on the market require a large number of inputs to function properly; if you only have a few responses, you may as well read responses by hand.

The more important reason for a large sample size is segmentation. Once you get the results of the original survey, you’ll probably have other questions. Imagine that you’re looking at visitor satisfaction with the site after a change. You notice that it has improved significantly overall, but there’s a large range in responses.

Should you be content with that data? Of course not. You should wonder whether there’s a hidden pattern to your responses—did men prefer the change, but not women? Did it work better for younger visitors, but not ...