Chapter 2. Building an Experimentation Platform
An experimentation platform is a critical part of your data infrastructure. This chapter takes a look at how to build a platform for scalable experimentation. Experimentation platforms consist of a robust targeting engine, a telemetry system, a statistics engine, and a management console.
In an experiment, users (the “experimental unit”) are randomly divided between two or more variants1 with the baseline variant called control, and the others called treatments. In a simple experiment, there are two variants: control is the existing system, and treatment is the existing system plus a new feature. In a well-designed experiment, the only difference between treatment and control is the feature. Hence, you can attribute any statistically significant difference in metrics between the variants to the new feature.
The targeting engine is responsible for dividing users across variants. It should have the following characteristics:
The engine should be fast to avoid becoming a performance bottleneck.
A user should receive the same variant for two different experiments only by chance.
A user is given the same variant for an experiment on each evaluation.
Here is a basic API for the targeting engine:
/** * Compute the variant for a (user, experiment). * * @param userId - a unique key representing the user * e.g., UUID, email * @param experiment - the name of the experiment * @param attributes - optional ...
Get Understanding Experimentation Platforms now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.