Practical A/B Testing

Creating Insights About Your Data

The first rule of using data to demonstrate the effect of changes in an A/B test is never to assume your data is always correct. To trust your test results and the experiment configuration, you’ll need to trust that your data is accurate. How do you do this in practice? Simple. Create mechanisms to monitor the state and overall quality.

To start, you could manually check the data, but you know that it isn’t scalable. The ideal solution would be a programmatic method that’s better than human eyeballs scanning petabytes of data. There are several ways to build programmatic validation:

• Implementing null checks in fields that ultimately should have non-null values.

• Computing the general distribution of event counts. ...