Future Savvy: Identifying Trends to Make Better Decisions, Manage Uncertainty, and Profit from Change

CHAPTER 2

The Quality of

Information: How

Good Is the Data?

This page intentionally left blank

41

A

FORECAST COMMUNICATES INFORMATION BE-

tween forecaster and reader and is subject to

the same standards of accuracy, truth-telling, and

bias-control by which one would judge any commu-

nication. As we have seen, forecasts can be very

different in methods and goals, but all forecasts lay

claim to factual truth. Future-aligning forecasts will

cite data in order to justify an expectation of a fu-

ture outcome. Future-influencing forecasts will cite

data to motivate the reader to support initiatives

and therein move the world. Optimists and pes-

simists will both justify themselves with data.

Therefore, the quality of the data and its inter-

pretation is our next consideration. When a fore-

cast brings data to support its hypothesis and

conclusions, we need to ask how diligently this data

has been researched and presented, whether it is appropriate and rel-

evant to sustain the point, and whether it has been fairly represented,

fairly used (is there conflicting evidence?), and fairly interpreted.

Poor forecasting starts with poor data or poor interpretation of data.

Good foresight work is inevitably tied to good facts about the status

quo and fair interpretation of trends from this. We cannot forecast ad-

equately unless we know where things are today and the evidence

of forces at work on today, which will lead to tomorrow.

This chapter deals with the various ways in which data, and par-

ticularly numerical data, can be less solid than it looks, even with

the best intentions. Chapter 3 deals with interpretative lenses that

the forecaster uses to choose, amalgamate, and understand the

data, including active misrepresentation and misinterpretation of

data, intentional bias and spin.

Why Numbers Aren’t as Solid as They Seem

We live in a world that values numbers and statistics highly. Num-

bers are our proxy for facts about the world; they allow us to think

about it in a concrete way and to measure the results of our actions

or inactions. Almost without exception, policy decisions of any sig-

nificance—from acting against greenhouse emissions to managing

school curricula—are made or backed up by numbers. We deter-

mine infant mortality ratios to tell us about social development; we

compute polar ice melting to judge global warming, and so on.

Where we enact policies to solve problems and advance welfare,

we measure success by seeing whether and by how much we have

moved the data. We measure, count, and compute to yield data

and we ask, What does the data say? What numbers would be

preferable? How do we achieve those numbers?

Similarly, we assess businesses by measurements such as return

on investment, product cycle times, or rate of “capital turns,” and

42

✧

FUTURE SAVVY

judge management by whether it pushes the numbers in the right

direction. Business success is judged by numbers even in areas

where numbers are hard to assess. The management “Balanced

Scorecard,” for example, pushes quantification deep into the non-

financial aspects of a company.

But numbers, and the facts they represent, are never purely ob-

jective. Any marketer will tell you that the wording of survey or

focus group questions greatly affects an-

swers, and this is the tip of the iceberg.

Every data point, however statistically

valid its construction, contains human

choices within it: choices about what is

looked for, what is counted, what it is

associated with, and how it is inter-

preted. Between one analyst and an-

other or one time and another, different

decisions will be made about what to observe, how to identify sig-

nificant instances, which ones to count, how to measure, and how

to calculate.

Moreover, data are seldom created or presented purely for our

benefit. They are more usually soldiers of advocacy, created or

chosen in order to be marched into battles over agendas and re-

sources. Purveyors of data very often selectively choose data points

to support their point of view or bring others around to it, to draw

attention away from a problem or toward a proposed solution, or

to arouse or diffuse public concern, and argue for a proposed in-

tervention or outcome.

1

As Joel Best, in his book Damned Lies and

Statistics—the best modern text on an old theme—says, “all statis-

tics are ammunition.” And not only are “hard” data subject to po-

litical interpretation and manipulation, but this interpretation is

often presented via the data to benefit from the natural “truth claim”

that numbers have.

CHAPTER 2: THE QUALITY OF INFORMATION

✧

43

Every data point, however

statistically valid its con-

struction, contains human

choices within it: choices

about what is looked for,

what is counted, what it is

associated with, and how it

is interpreted.