April 2018
Intermediate to advanced
30 pages
36m
English
Content preview from Understanding Experimentation Platforms
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
Chapter 3. Designing Metrics
An experiment is a way to test the impact, if any, that a feature (or a variation of a feature) has on a metric. In this chapter, we take a deeper dive into the world of metrics. Our focus is on describing different types of metrics and qualities of good metrics, qualities of good metrics, as well as a taxonomy for metrics. For a more in-depth look, we refer you to this research paper from Microsoft1.
Types of Metrics
Four types of metrics are important to experiments. The subsections that follow take a look at each one.
Overall Evaluation Criteria
Overall Evaluation Criteria (OEC) is the metric that an experiment is designed to improve. For example, rides per user and reviews per business are good OEC’s for Uber and Yelp, respectively. As a rule of thumb, value per experimental unit is an OEC formula, where value is experienced by the experimental unit as a result of the service.
The OEC is a measure of long-term business value or user satisfaction. As such, it should have three key properties:
- Sensitivity
-
The metric should change due to small changes in user satisfaction or business value so that the result of an experiment is clear within a reasonable time period.
- Directionality
-
If the business value increases, the metric should move consistently in one direction. If it decreases, the metric should move in the opposite direction.
- Understandability
-
OEC ties experiments to business value, so it should be easily understood by business executives. ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.