280 ◾ PRAGMATIC Security Metrics
(by the numbers). However, information security is a branch of risk management.
Even with the world’s greatest information security measurement system comprising
a suite of the most PRAGMATIC security metrics, we must not ignore the funda-
mental fact that we are dealing with things that are, to varying extents, inherently
We might be able to tame information security risk, but we will never domesticate it.
As a consequence, there are inherent unpredictabilities with some information
security metrics. We can do our level best to minimize them by using better, more
reliable instrumentation and to smooth them out using the statistical techniques
described in the next chapter, but they inevitably remain.
Sometimes someone with suﬃcient experience in the area—you, perhaps—
may feel the numbers simply don’t add up: the metrics indicate a particular course
of action that, for some reason, your experience tells you is not appropriate. Such
discrepancies may be distinctly unsettling at the time but can be fascinating to
examine in more detail:
◾ Is it that the numbers truly are lying, perhaps because the raw data are wrong
or the analysis is faulty?
◾ Are the numbers misleading because they don’t take suﬃcient account of
all the relevant factors (meaning the model or framework underpinning the
metrics is ﬂawed)?
◾ Has something changed, so the metrics no longer make sense?
◾ Or is that the numbers and analysis are, in fact, correct, but for once, your
gut feeling has let you down?*
If the information security measurement system is generally sound enough to have
proven itself trustworthy, the latter conclusion may be management’s default pre-
sumption in the absence of compelling evidence that something else is to blame. In
other words, management may need to be convinced not to do whatever the metrics
are suggesting but to follow a diﬀerent course.
e issue of trusting your metrics is superbly demonstrated by experiments per-
formed by the U.S. Navy way back in the 1940s brought about by an increasing
number of aircraft accidents in zero-visibility conditions, such as ﬂying into clouds
or heavy fog. It turned out that even the best pilots were unable to ﬂy straight and
level for less than half a minute using just their senses. is led to the development
of instrumentation that is used today for ﬂight under those conditions, known as
IFR or instrument ﬂight rules. Acquiring an instrument ticket is typically the most
arduous licensing, and the requisite is absolute trust in the instruments—properly
Such is the route to enlightenment, Grasshopper.
is is a distinctly dangerous route to take, especially if the measurement system is mature, as
it devalues and may even discredit the measurement system in the eyes of the more cynical and
thoughtful managers anyway.