Practical A/B Testing

Introducing the Non-inferiority A/B Test

At this point, we love testing things. We love learning. We love proving that a new feature is better than the existing user experience. But sometimes, we may not need to demonstrate that something is better than the control but rather that something is minimally worse or as good as it. In these cases, the non-inferiority test is the best approach.

For a non-inferiority A/B test, you can conclude that your change did not worsen a key product or business metric more than a predefined level. The goal is to learn whether the change is performing equal to or slightly worse than the existing, unchanged control experience. Here, slightly worse is a specific margin that defines the inferior threshold.

To illustrate ...