A/B Testing: The Most Powerful Way to Turn Clicks Into Customers

CHAPTER11

How A/B Tests Can Go Awry

Potential Mistakes and Pitfalls to Avoid

At this point in the book the hazards of not A/B testing should be clear; however, it's worth mentioning that A/B testing has its own set of pitfalls and “worst-practices.”

Testing without Traffic

The good news is that you need only two things to conduct an A/B test: a website with some content on it, and visitors. The more traffic you have the faster you will see statistically significant results about how each variation performed.

What A/B testing your site can't do, however, is generate that traffic in the first place. A blogger who's just getting off the ground and has only 100 visitors per month would be better off focusing primarily on content and building a following of users (bolstered perhaps by SEO or paid ads) who provide traffic to the site before delving into the statistics of optimizing that traffic. After all, you have to generate the traffic in the first place before you do anything with it. (In addition, in a site's fledgling period, a handful of conversations with real users will offer more feedback than you will get from an A/B test on a sparsely trafficked site.) While optimization can help even the smallest enterprise, it's also true that testing becomes faster, more precise, and more profitable the more user traffic you have to work with.

The Beginning of the Funnel versus the End: UserVoice

Testing will occasionally reveal a change that increases one metric but decreases another. ...