A/B Testing: The Most Powerful Way to Turn Clicks Into Customers

CHAPTER3

Seek the Global Maximum

Refinement and Exploration

Premature optimization is the root of all evil.

Donald Knuth

Imagine you're climbing a mountain: if your goal is to get to the top of the tallest mountain, and you don't have a map of the range, it's probably not a good idea just to start walking up the nearest slope. You'll climb, and climb, and then ultimately reach some peak—and then what? Where do you move next if this peak doesn't turn out to be the highest one?

In optimization, the term for the nearby, uphill peak is the local maximum, whereas the distant, largest peak is the global maximum.

One of the things that we like to tell companies that we work with is to be willing to think big. Being too complacent about the status quo can lead to focusing too much on fine-tuning. As Figure 3.1 highlights, the “Refinement” path might lead you to miss out on the best solution that could have been discovered with the “Exploration” approach. While refinement can lead to a solution better than what you have today, we recommend exploring multiple alternatives that might not resemble the current site first. We encourage the kind of humility and bravery required to say, “You know, the website we have today is far from perfect. Let's try some dramatically new layouts, new designs, and redesigns, figure out which of those work well, and then refine from there.”

However, it's not as simple as saying that one should always explore first and always refine second. The truth is that ...