The Quality of
Information: How
Good Is the Data?
This page intentionally left blank
tween forecaster and reader and is subject to
the same standards of accuracy, truth-telling, and
bias-control by which one would judge any commu-
nication. As we have seen, forecasts can be very
different in methods and goals, but all forecasts lay
claim to factual truth. Future-aligning forecasts will
cite data in order to justify an expectation of a fu-
ture outcome. Future-influencing forecasts will cite
data to motivate the reader to support initiatives
and therein move the world. Optimists and pes-
simists will both justify themselves with data.
Therefore, the quality of the data and its inter-
pretation is our next consideration. When a fore-
cast brings data to support its hypothesis and
conclusions, we need to ask how diligently this data
has been researched and presented, whether it is appropriate and rel-
evant to sustain the point, and whether it has been fairly represented,
fairly used (is there conflicting evidence?), and fairly interpreted.
Poor forecasting starts with poor data or poor interpretation of data.
Good foresight work is inevitably tied to good facts about the status
quo and fair interpretation of trends from this. We cannot forecast ad-
equately unless we know where things are today and the evidence
of forces at work on today, which will lead to tomorrow.
This chapter deals with the various ways in which data, and par-
ticularly numerical data, can be less solid than it looks, even with
the best intentions. Chapter 3 deals with interpretative lenses that
the forecaster uses to choose, amalgamate, and understand the
data, including active misrepresentation and misinterpretation of
data, intentional bias and spin.
Why Numbers Aren’t as Solid as They Seem
We live in a world that values numbers and statistics highly. Num-
bers are our proxy for facts about the world; they allow us to think
about it in a concrete way and to measure the results of our actions
or inactions. Almost without exception, policy decisions of any sig-
nificance—from acting against greenhouse emissions to managing
school curricula—are made or backed up by numbers. We deter-
mine infant mortality ratios to tell us about social development; we
compute polar ice melting to judge global warming, and so on.
Where we enact policies to solve problems and advance welfare,
we measure success by seeing whether and by how much we have
moved the data. We measure, count, and compute to yield data
and we ask, What does the data say? What numbers would be
preferable? How do we achieve those numbers?
Similarly, we assess businesses by measurements such as return
on investment, product cycle times, or rate of “capital turns,” and
judge management by whether it pushes the numbers in the right
direction. Business success is judged by numbers even in areas
where numbers are hard to assess. The management “Balanced
Scorecard,” for example, pushes quantification deep into the non-
financial aspects of a company.
But numbers, and the facts they represent, are never purely ob-
jective. Any marketer will tell you that the wording of survey or
focus group questions greatly affects an-
swers, and this is the tip of the iceberg.
Every data point, however statistically
valid its construction, contains human
choices within it: choices about what is
looked for, what is counted, what it is
associated with, and how it is inter-
preted. Between one analyst and an-
other or one time and another, different
decisions will be made about what to observe, how to identify sig-
nificant instances, which ones to count, how to measure, and how
to calculate.
Moreover, data are seldom created or presented purely for our
benefit. They are more usually soldiers of advocacy, created or
chosen in order to be marched into battles over agendas and re-
sources. Purveyors of data very often selectively choose data points
to support their point of view or bring others around to it, to draw
attention away from a problem or toward a proposed solution, or
to arouse or diffuse public concern, and argue for a proposed in-
tervention or outcome.
As Joel Best, in his book Damned Lies and
Statistics—the best modern text on an old theme—says, “all statis-
tics are ammunition.” And not only are “hard” data subject to po-
litical interpretation and manipulation, but this interpretation is
often presented via the data to benefit from the natural “truth claim”
that numbers have.
Every data point, however
statistically valid its con-
struction, contains human
choices within it: choices
about what is looked for,
what is counted, what it is
associated with, and how it
is interpreted.

Get Future Savvy: Identifying Trends to Make Better Decisions, Manage Uncertainty, and Profit from Change now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.