Understanding Experimentation Platforms

Chapter 2. Building an Experimentation Platform

An experimentation platform is a critical part of your data infrastructure. This chapter takes a look at how to build a platform for scalable experimentation. Experimentation platforms consist of a robust targeting engine, a telemetry system, a statistics engine, and a management console.

Targeting Engine

In an experiment, users (the “experimental unit”) are randomly divided between two or more variants¹ with the baseline variant called control, and the others called treatments. In a simple experiment, there are two variants: control is the existing system, and treatment is the existing system plus a new feature. In a well-designed experiment, the only difference between treatment and control is the feature. Hence, you can attribute any statistically significant difference in metrics between the variants to the new feature.

The targeting engine is responsible for dividing users across variants. It should have the following characteristics:

Fast

The engine should be fast to avoid becoming a performance bottleneck.

Random

A user should receive the same variant for two different experiments only by chance.

Sticky

A user is given the same variant for an experiment on each evaluation.

Here is a basic API for the targeting engine:

/** * Compute the variant for a (user, experiment). * * @param userId - a unique key representing the user * e.g., UUID, email * @param experiment - the name of the experiment * @param attributes - optional ...