Behavioral Data Analysis with R and Python

Part IV. Designing and Analyzing Experiments

Running experiments is the bread and butter of behavioral scientists and causal data scientists in business. Indeed, randomizing the allocation of subjects between experimental groups allows us to negate any potential confounding without the need to even identify it.

Books about A/B testing abound. How is the presentation in this one different? I would argue that several aspects of its approach make it both simpler and more powerful.

First of all, recasting experiments within the causal-behavioral framework will allow you to create better and more effective experiments, and better understand the spectrum from observational to experimental data analysis instead of thinking of them as separate.

Second, most books on A/B testing rely on statistical tests such as the T-test of means or the test of proportions. Instead, I’ll rely on our known workhorses, linear and logistic regressions, which will make our experiments simpler and more powerful.

Finally, traditional approaches to experimentation decide whether to implement the tested intervention based on its p-value, which doesn’t lead to the best business decisions. Instead I’ll rely on the Bootstrap and its confidence intervals, which is progressively establishing itself as the best practice.

Therefore, Chapter 8 will show what a “simple” A/B test looks like when using regression and the Bootstrap. That is, for each customer we toss a metaphorical coin. Heads and they see version A, tails ...