Creating a Data-Driven Organization

Chapter 8. A/B Testing

He who consistently uses #abtesting to make data-driven decisions has invariably been humbled by the poor success rate of ideas.

Ron Kohavi

I’ve come to terms with the fact that the point of experiments, data, and testing, isn’t for me to be right…I needed the information from these tests to get to the right answer, eventually.

PJ McCormick¹

It was 1998, and Greg Linden, one of Amazon’s early engineers, had an idea. Why not create recommendations on checkout? Supermarkets put candy at the checkout aisle to stimulate impulse buys. That works. Why not peek into the Amazon.com cart and make personalized, relevant recommendations that the customer might appreciate? He hacked up a prototype, got it working, and showed it around. The rest of the story is best told in his own words:

While the reaction was positive, there was some concern. In particular, a marketing senior vice-president was dead set against it. His main objection was that it might distract people away from checking out—it is true that it is much easier and more common to see customers abandon their cart at the register in online retail—and he rallied others to his cause.

At this point, I was told I was forbidden to work on this any further. I was told Amazon was not ready to launch this feature. It should have stopped there.

Instead, I prepared the feature for an online test. I believed in shopping cart recommendations. I wanted to measure the sales impact.

I heard the SVP was angry when he ...