The Ways to New

Chapter 10Data-Driven Disruption

Take a candy jar. One of those giant clear glass candy jars with a big, flat, round stopper you see in old-fashioned candy stores. It has got hundreds, maybe thousands of pieces of candy in it. Someone asks, “So how many pieces of candy does the jar contain?” The chances are you won't get the right answer. You will be wrong by a wide margin. But if lots of people are asked the same question and an average of their answers is taken, the figure reached will prove astonishingly close. At first sight this seems more than improbable. Statisticians, though, are familiar with the principle. It is called “The Wisdom of Crowds.”

In 2013, one of our agencies in Australia applied this to predicting the outcome of the Melbourne Cup, a race that brings together the best thoroughbreds in the world. We collated all the tweets we could find about all the horses in the race, as well as their trainers and jockeys. We recorded all the favorable comments we could find, tens of thousands of them, about each of the entries. This process produced what you might call an Internet favorite, a horse called “Fiorente.” The bookies' odds on Fiorente were not all that hot. The horse was far from favorite.

It won the race.

Statistical evidence of this sort shows that we all share in some kind of collective, subterranean intelligence. Data, you could say, teaches us how to apprehend reality. Utilizing data increases our chances of success in any undertaking. And this fact opens ...