Chapter 7
Metrics for Decision
The Need forCausal Models
    soware metrics is to help soware profes-
sionals (be they developers, testers, managers, or maintainers) make
decisions under uncertainty. As explained so far, we are ultimately inter-
ested in knowing things like how much eort and time will be required to
produce a system to a particular set of requirements, how much more test-
ing will be needed before the system has suciently few bugs, how much
eort can we save by using some particular tool, etc. All of these deci-
sions involve uncertainty, risk, and trade-os. Whereas the other chapters
in this book provide the tools for identifying and collecting the specic
metrics to capture the underlying attributes involved in these problems,
this chapter focuses on how to incorporate these metrics into a rational
decision-making framework.
Several years ago at a leading international soware metrics confer-
ence, which attracted nearly a thousand delegates, a keynote speaker
recounted an interesting story about a company-wide metrics program
that he had been instrumental in setting up. He said that one of the main
objectives of the program was to achieve process improvement by learn-
ing from metrics what process activities worked and what ones did not.
is chapter is based on collaborative work with Martin Neil.
292 Software Metrics
To do this, the company looked at those projects that, in metrics terms,
were considered most successful. ese were the projects with especially
low rates of customer-reported defects, measured by defects per thousand
lines of code (KLOC). e idea was to learn what processes characterized
such successful projects. A number of such “star” projects were identied,
including some that achieved the magical perfect quality target of zero
defects per KLOC in the rst six months post-release. But, it turned out
that what they learned from this was very dierent from what they had
expected. Few of the star projects were, in fact, at all successful from any
commercial or subjective perspective:
e main explanation for the very low number of defects reported
by customers was that they were generally so poor that they never
got properly completed or used.
is story exposes the classic weakness of relying only on traditional
static metrics: the omission of sometimes obvious and simple causal fac-
tors that can have a major explanatory eect on what is observed and
learnt. If you are managing a soware development project and you hear
that very few defects were found during a critical testing phase, is that
good news or bad news? Of course it depends on the testing eort, just as
it does if you heard that a large number of defects were discovered. e
danger is to assume that defects found in testing correlates with defects
found in operation, and that it is thus possible to build a regression model
(in the sense of Chapter 6) in which defects in operation (the uncertain
quantity we are trying to predict) is a function of the variable defects found
in test (the known quantity we can measure). Similarly, it is dangerous to
assume that we can build regression models in which other metrics, such
as size and complexity metrics, act as independent variables that can pre-
dict dependent variables like defects and maintenance eort. In Section
7.1, we will explain why such attempts at prediction based on correlation
are not normally eective and cannot be used for risk assessment.
If you are interested in, say, predicting defects in operation from infor-
mation gained during development and testing, then certainly metrics
like defects found in test and other size and complexity metrics might be
useful pieces of evidence. A rational way to use this evidence to update
your beliefs about defects in operation is to use Bayesian reasoning. When
there are multiple, dependent metrics then we use Bayesian reasoning
together with causal models, that is, Bayesian networks (BNs). We explain
the basics of Bayesian Reasoning and BNs in Section 7.2. In Section 7.3,

Get Software Metrics, 3rd Edition now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.