Data Science: The Hard Parts

Chapter 16. A/B Tests

Chapter 15 described the importance of randomization to estimate causal effects, when this option is actually available to the data scientist. A/B tests use this power to improve an organization’s decision-making capabilities in a process analogous to local optimization.

This chapter describes A/B tests and should help you navigate the many intricacies of a relatively simple procedure for improved decision making.

What Is an A/B Test?

In its simplest form, an A/B test is a method to evaluate which one of two alternatives is better in terms of a given metric. A denotes the default or baseline alternative, and B is the contender. More complex tests can present several alternatives at the same time to find the best one. Using the language from Chapter 15, units that get A or B are also called control and treatment groups, respectively.

From this description you can see that there are several ingredients in every A/B test:

Metric

Being at the heart of improved decision making, the design of A/B tests should always start by choosing the right metric. The techniques described in Chapter 2 should help you find a suitable metric for the test you want to implement. I’ll denote this outcome metric with $Y$.

Levers or alternatives

Once you define a metric, you can go back and think of the levers that most directly affect it. A common mistake is to start with an alternative (say, the background color of a button in your web page or app) and try to reverse engineer ...