Understanding Experimentation Platforms

Chapter 3. Designing Metrics

An experiment is a way to test the impact, if any, that a feature (or a variation of a feature) has on a metric. In this chapter, we take a deeper dive into the world of metrics. Our focus is on describing different types of metrics and qualities of good metrics, qualities of good metrics, as well as a taxonomy for metrics. For a more in-depth look, we refer you to this research paper from Microsoft¹.

Types of Metrics

Four types of metrics are important to experiments. The subsections that follow take a look at each one.

Overall Evaluation Criteria

Overall Evaluation Criteria (OEC) is the metric that an experiment is designed to improve. For example, rides per user and reviews per business are good OEC’s for Uber and Yelp, respectively. As a rule of thumb, value per experimental unit is an OEC formula, where value is experienced by the experimental unit as a result of the service.

The OEC is a measure of long-term business value or user satisfaction. As such, it should have three key properties:

Sensitivity

The metric should change due to small changes in user satisfaction or business value so that the result of an experiment is clear within a reasonable time period.

Directionality

If the business value increases, the metric should move consistently in one direction. If it decreases, the metric should move in the opposite direction.

Understandability

OEC ties experiments to business value, so it should be easily understood by business executives. ...