Practical A/B Testing

Defining Test and Control Variants

A variant, also called a treatment or segment, defines the subset of users from the target population. An A/B test consists of the following:

• A test variant in which users receive a different product experience or functionality that the team is looking to evaluate.

• A control variant in which users receive the existing or unchanged functionality.

Your test and control variants should consist of users who:

• Meet the requirements to be qualified for the experiment.

• Have been selected at random to prevent sampling errors or to avoid creating conclusions based on a partially represented user population.

• Have no data-logging restrictions so that product engagement and interactions are available in the data sets ...