Understanding Experimentation Platforms

Chapter 1. Introduction

Engineering agility has been increasing by orders of magnitude every five years, almost like Moore’s law. Two decades ago, it took Microsoft two years to ship Windows XP. Since then, the industry norm has moved to shipping software every six months, quarter, month, week—and now, every day. The technologies enabling this revolution are well-known: cloud, Continuous Integration (CI), and Continuous Delivery (CD) to name just a few. If the trend holds, in another five years, the average engineering team will be doing dozens of daily deploys.

Beyond engineering, Agile development has reshaped product management, moving it away from “waterfall” releases to a faster cadence with minimum viable features shipped early, followed by a rapid iteration cycle based on continuous customer feedback. This is because the goal is not agility for agility’s sake, rather it is the rapid delivery of valuable software. Predicting the value of ideas is difficult without customer feedback. For instance, only 10% of ideas shipped in Microsoft’s Bing have a positive impact.¹

Faced with this fog of product development, Microsoft and other leading companies have turned to online controlled experiments (“experiments”) as the optimal way to rapidly deliver valuable software. In an experiment, users are randomly assigned to treatment and control groups. The treatment group is given access to a feature; the control is not. Product instrumentation captures Key Perfomance Indicators (KPIs) ...