Analytics Stories

CHAPTER 49Can A/B Testing Improve My Website's Performance?

If your company has a website, there are many decisions you need to make. A/B testing (sometimes called split testing) allows a website to test whether changes in a website will increase revenue, click-through rates, or other metrics. The basic idea is simple and is a special case of randomized controlled trials (RCTs). Call the changed website design the Challenger and the current website design the Status Quo. Table 49.1 shows some examples of commonly considered changes in website design for a shopping site.

Table 49.1: Examples of A/B tests

STATUS QUO	CHALLENGER
Purchase button is on upper-left portion of page.	Put purchase button on upper-right portion of page.
Purchase button is red.	Make purchase button blue.
Show a picture of product.	Show a video of a person using the product.

If desired, you can compare more than two alternatives. For example, you could compare how red, blue, and green purchase buttons influence the chance that a potential customer purchases the product.

To use A/B testing for comparing two alternatives, you create two versions of your site at the same URL: one version with the Status Quo and one version with the Challenger. Each visitor to your URL has a 50% chance of being sent to each version of the site. The randomization ensures that the potential customers sent to each version should be virtually identical on all attributes influencing the likelihood of purchase, ...