O'Reilly logo

PRAGMATIC Security Metrics by W. Krag Brotby, Gary Hinson

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

279
Chapter 10
Downsides of Metrics
PRAGMATIC
security metrics
Chapter 1
Introduction
Chapter 13
Conclusion
Chapter 2
Why measure?
Chapter 12
Case study
Chapter 3
Art and science
Chapter 11
Using metrics
Chapter 4
Audiences
Chapter 10
Downsides
Chapter 5
Finding metrics
Chapter 9
Advanced metrics
Chapter 6
PRAGMATIC
Chapter 8
Measurement
System
Chapter 7
Example metrics
Appendices
Politics
Implausible
deniability
Gaps
What not
to measure
But everything you do in life has a downside.
Melissa Auf der Maur
is chapter acknowledges that although there are tremendous advantages to selecting
and using appropriate information security metrics, there are also a few drawbacks.
10.1 Numbers Dont Always Tell the Whole Story
Be careful what you wish for! A suite of PRAGMATIC security metrics, particularly
within an information security measurement system, will provide the information
management needs to manage information security to a large extent scientifically
280 ◾  PRAGMATIC Security Metrics
(by the numbers). However, information security is a branch of risk management.
Even with the world’s greatest information security measurement system comprising
a suite of the most PRAGMATIC security metrics, we must not ignore the funda-
mental fact that we are dealing with things that are, to varying extents, inherently
unpredictable.
We might be able to tame information security risk, but we will never domesticate it.
As a consequence, there are inherent unpredictabilities with some information
security metrics. We can do our level best to minimize them by using better, more
reliable instrumentation and to smooth them out using the statistical techniques
described in the next chapter, but they inevitably remain.
Sometimes someone with sufficient experience in the area—you, perhaps—
may feel the numbers simply dont add up: the metrics indicate a particular course
of action that, for some reason, your experience tells you is not appropriate. Such
discrepancies may be distinctly unsettling at the time but can be fascinating to
examine in more detail:
Is it that the numbers truly are lying, perhaps because the raw data are wrong
or the analysis is faulty?
Are the numbers misleading because they dont take sufficient account of
all the relevant factors (meaning the model or framework underpinning the
metrics is flawed)?
Has something changed, so the metrics no longer make sense?
Or is that the numbers and analysis are, in fact, correct, but for once, your
gut feeling has let you down?*
If the information security measurement system is generally sound enough to have
proven itself trustworthy, the latter conclusion may be management’s default pre-
sumption in the absence of compelling evidence that something else is to blame. In
other words, management may need to be convinced not to do whatever the metrics
are suggesting but to follow a different course.
e issue of trusting your metrics is superbly demonstrated by experiments per-
formed by the U.S. Navy way back in the 1940s brought about by an increasing
number of aircraft accidents in zero-visibility conditions, such as flying into clouds
or heavy fog. It turned out that even the best pilots were unable to fly straight and
level for less than half a minute using just their senses. is led to the development
of instrumentation that is used today for flight under those conditions, known as
IFR or instrument flight rules. Acquiring an instrument ticket is typically the most
arduous licensing, and the requisite is absolute trust in the instruments—properly
*
Such is the route to enlightenment, Grasshopper.
is is a distinctly dangerous route to take, especially if the measurement system is mature, as
it devalues and may even discredit the measurement system in the eyes of the more cynical and
thoughtful managers anyway.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required