7. On Analysis

Trusting your findings

Understanding the results of your experiments can be a complicated affair. Luckily, there are a few simple methods to make sense of all the data your experiments accumulate and give you confidence in your findings.

Statistical significance

In your experiments, you attempt to hold all variables constant except the thing you want to investigate. The reality, of course, is that many variables are not constant. You’re comparing similar groups, but those groups themselves will have differences that introduce noise. That’s where statistical significance comes in (Figure 7.1).


Figure 7.1. Statistical significance ...

Get Experiment!: Website conversion rate optimization with A/B and multivariate testing now with the O’Reilly learning platform.

O’Reilly members experience live online training, plus books, videos, and digital content from nearly 200 publishers.