Understanding Experimentation Platforms

Chapter 5. Common Pitfalls

Pitfalls have the potential to live in any experiment. They begin with how users are assigned across variants and control, how data is interpreted, and how metrics impact is understood.

Following are a few of the most common pitfalls to avoid.

Sample Ratio Mismatch

The meaningful analysis of an experiment is contingent upon the independent and identical distribution of samples between the variants.

If the samples are not selected truly at random, any conclusions drawn can be attributable to the way the samples were selected and not the change being tested. You can detect a sampling bias in your randomization by ensuring that the samples selected by your targeting engine match the requested distribution within a reasonable confidence interval.

If you design an experiment with equal percentages (50/50), and the actual sample distribution varies from the expected ratio, albeit small, the experiment might have an inherent bias, rendering the experiment’s results invalid. The scale of this deviation should shrink as your sample sizes increase. In the case of our 50/50 rollout, the 95th percent confidence interval for 1,000 samples lies at 500 ± 3.1%. Simply put, with 1,000 users, if you have more than 531 users in any given treatment, you have a sample ratio mismatch (SRM). This delta shrinks as the sample size increases. With 1,000,000 samples a variation of ±0.098% in the sample distribution would be cause for concern. As you can see it is important ...