A/B Testing: The Most Powerful Way to Turn Clicks Into Customers

CHAPTER10

Iterate, Iterate, Iterate

The Art of Asking Many Small Questions Rather than One Big One

When test results first start to come in, it's important to recognize that they're likely going to generate more questions than answers. These questions will likely also point to more assumptions you find you want to test. That's okay; in fact, that's how it's supposed to work. And that's the perfect starting point for your tests to come.

One of the biggest questions that we had when starting Optimizely was whether people would continue experimenting after they found a “local maximum,” that is, the best set of tweaks for their current design or current funnel. We worried that people would spend a few months optimizing a site—a better headline here, different image there—and be done with it. We didn't want businesses to think of A/B testing as a finite, onetime process, and we acknowledged the disconcerting possibility that businesses would feel satisfied if the site worked better than before, and might throw in the towel and declare, “Well, enough of that! We're all done A/B testing.”

Much to our delight, we've found that the opposite has occurred. After a few big wins, most people realize that they're just beginning the testing journey—and the success they've enjoyed propels them to keep going.

In month one, you run a couple of tests; you get some wins and you're adding those wins into the production code. Those wins might have been scattered around different pages of your site, ...