Behavioral Data Analysis with R and Python

Chapter 8. Experimental Design: The Basics

Let’s start our exploration of experimentation with a very simple experiment: influenced by a leading online store, AirCnC’s management has decided that a “1-click booking” button is just what’s needed to boost AirCnC’s booking rate. As I discussed earlier, we’ll assign customers to our experimental groups one by one as they connect to the website. This is the simplest possible type of experiment, and many companies offer interfaces that allow you to create and start running A/B tests like this in a matter of minutes.

This straightforward experiment will be the opportunity to go through the process without getting bogged down in technical considerations:

1. The first step is to plan the experiment. This is where the causal-behavioral perspective comes in, to help ensure that you have clearly defined criteria for success, and that you understand what it is you’re testing and how you expect it to impact your target metric.

2. Then, after reviewing the data and packages that we’ll use in the rest of the chapter, I’ll show you how to do the random assignment and determine the sample size for your experiment.

3. Finally, we’ll analyze the results of the experiment, which in such a simple case will be very quick.

Note

The vocabulary of experimental design owes much to its statistical and scientific roots. I’ll talk of “control” and “treatment” groups as well as “interventions,” which may sound ominous or like overkill when we’re really discussing ...