What to Test
Optimization in Five Steps
The hardest part of A/B testing is determining what to test in the first place. Having worked with thousands of customers who do A/B testing every day, one of the most common questions we hear is, “Where do I begin?”
A mistake that some companies make is to start moving a bunch of levers around without clear planning upfront for what they're trying to optimize—and what will be impacted by those changes. It's tempting to just dive in and start changing parts of your homepage, or your product page, or your checkout page, without truly understanding the value that it's generating (or not generating) for your business.
Instead, we advise a purposeful and deliberate five-step process:
Step One: Define success
Step Two: Identify bottlenecks
Step Three: Construct a hypothesis
Step Four: Prioritize
Step Five: Test
This process begins with the most important question of all: What is the purpose of your site?
Step One: Define Success
Before you can determine which of your test's variations is the winner, you have to first decide how you're keeping score. To start A/B testing successfully, you need to answer a specific question: What is your website for? If you could make your website do one thing better, what would it do?
If the answer to that question isn't completely clear to you, there's a trick that might help. Imagine the following dialogue:
ALICE: “What do you want to achieve with A/B testing?”
BOB: “We don't know. We don't know ...