Elsevier UK Jobcode:RTF Chapter:CH11-H8304 22-3-2007 5:11p.m. Page:211 Trimsize:165×234MM
Fonts used:Ocean Sans & Sabon Margins:Top:42pt Gutter:54pt Font Size:10/12 Text Width:30p6 Depth:47 Lines
Testing, backtesting, post-mortems and experimental methodology 211
experimentally. In these and other firms, where internal prediction seems to have
given good results is in:
Capturing employee sentiment on project deadlines, and
Pinpointing problems in product quality more accurately than other measures
Associated with issues related to model review and analysis, which have been
brought to the reader’s attention in this section, should be tests designed for qualitative
factors which are not part of the algorithm, but can have significant impact on final
results. This enlarges the scope of stress testing a financial model, and brings into the
picture the issue of direct top management responsibility.
11.5 Post-mortems, experimentation and war games
Post-mortems have proved an invaluable tool in testing strategic decisions, from
investments to trades and the control of exposure. They provide significant informa-
tion on what went right and wrong including the original decision.
Johnson & Johnson has a policy, and a system, which sees to it that two years after
the company made a big acquisition senior management would do a thorough post-
mortem examination. This involves the top of the company’s hierarchy and takes the
form of an executive audit.
The post-mortem session starts with the original projections and original reasons
for the deal.
Participants identify the executives who made the arguments for acquisition and
compare these arguments with how the deal worked out.
It is very instructive to perform post-mortems on all types of major projects as
long as one does not get carried away with them. The goal is not witch-hunting, but
lessons to be learned and identification of errors not to be repeated. The ‘no blames’
policy is important because most managers don’t like doing post-mortems on their
key investment decisions because they are haunted by them.
There is always an excuse for something that went wrong, and many senior man-
agers spend a lot of time telling their board how wonderful their projects were, and
how justified their capital expenditures should have been. But they did not turn out
that way, and a post-mortem is the best way of taking the proverbial long, hard look
at what brought different results, after these results are known.
Chief executives renowned for sound governance behave differently, and for good
reason. They require that hours are spent examining stupid blunders, including the
reasons for underperformance and opportunity costs. They look where the institution
they are in charges, makes or loses too much money.
Which trading activity?
Which business unit?
Which elements turned the tables?
Was this adverse reaction foreseen?