The Flaw of Averages: Why We Underestimate Risk in the Face of Uncertainty

CHAPTER 20

Taking Credit for Chance Occurrences

You have probably heard the one about the marketing director who knew that half of her advertising dollars were effective; she just never knew which half. It’s hard enough to manage a business when you can directly measure the results of your actions. But when the things under your control are combined with numerous things beyond your control, such as market conditions, competition, and the weather, running a business is like shooting in the dark.

Advertising Results

Suppose, for example, that the sales figures have just come in from your marketing test. Of your 30 sales regions, you ran ads in ten, while the other 20 received no ads. The results consist of the percentage change in sales from the previous month for each market, as displayed in Table 20.1.¹

The ten markets in which you advertised saw sales increase by 7.35 percent on average, while the 20 markets without ads had sales grow on average by 3.93 percent. The difference in averages is 3.42 percent, so the ads obviously worked.

Or could the average difference of 3.42 percent have happened by chance? The eleventh deadly sin is taking credit for something that was in fact just the luck of the draw. This is the realm of the Red concept of HYPOTHESIS TESTING, which consists of things like the F-TEST and T-TEST. But the operable question is, “Did it happen by chance?”

TABLE 20.1 Percentage Change in Sales from Previous Month by Market

A simple example involves coin tosses. ...