97 Things Every Data Engineer Should Know

Chapter 2. A/B and How to Be

Sonia Mehta

At its core, A/B testing is a method of comparing two versions of something, to see which one performs better. A very simple example of this is adjusting the location of an ecommerce site’s shopping cart image from the top right to the bottom right. Perhaps some team members believe moving it to the bottom will lead to fewer abandoned carts. Depending on the size and nature of the experiment, data engineering may be involved with everything from the instrumentation and tracking to the analysis.

Related to this topic, it’s important to know that third-party tools exist to help set up backend tracking for experiments. Whether a third-party tool or an in-house solution is used, it’s critical to validate the results and feel comfortable about the experiment logging.

In validating experiment metrics, nuances will always need further investigating. The list can be quite long, but areas that can allow you to quickly detect an issue include the following:

Sample sizes

If the experiment is a 50/50 split, the sample sizes should be very close to one another. If the experiment is another split, validate the sample size against the expected weight.

Start and stop dates (with any ramp-up weights)

The experiment may have been slowly rolled out in a ladder, from 1%, 5%, 10%, etc., to avoid a potentially large adverse impact. The experiment may have also ...