Understanding Experimentation Platforms

Chapter 4. Best Practices

In this chapter, we look at a few best practices to consider when building your experimentation platform.

Running A/A Tests

To ensure the validity and accuracy of your statistics engine, it is critical to run A/A tests. In an A/A test, both the treatment and control variants are served the same feature, confirming that the engine is statistically fair and that the implementation of the targeting and telemetry systems are unbiased.

When drawing random samples from the same distribution, as we do in an A/A test, the p-value for the difference in samples should be distributed evenly across all probabilities. After running a large number of A/A tests, the results should show a statistically significant difference exists at a rate that matches the platform’s established acceptable type I error rate (α).

Just as a sufficient sample size is needed to evaluate an experimental metric, so too does the evaluation of the experimentation platform require many A/A tests. If a single A/A test returns a false positive, it is unclear whether this is an error in the system or if you simply were unlucky. With a standard 5% α, a run of 100 A/A tests might see somewhere between 1 and 9 false positives without any cause for alarm.

There could be a number of reasons for failing the A/A test suite. For example, there could be an error in randomization, telemetry, or stats engine. Each of these components should have their debugging metrics to quickly pinpoint the source of ...