78 9. EXPERIMENTAL VALIDATION AND ANALYSIS
In the case of database locking, for example, should a transactions own locks be taken into
account in calculating the probability of a conflict? Not having to do so may greatly simplify the
analysis. One could make the choice by developing the model both ways, and using simulation
measurements to see if the simplification causes a significant loss in accuracy.
9.2.2 DATA PRESENTATION
In presenting experimental data to validate a model, one should bear in mind the approximations
in the model and the accuracy of the experiment. For example, it is not meaningful to use multiple
decimal places when such precision is unwarranted.
It is often easier to compare numerical values with a data plot. However, one’s judgment
can be skewed if the axes are translated or scaled. For instance, a vertical axis that does not start
at 0 can exaggerate the performance difference between two algorithms, and a logarithmic axis
can understate the difference between model prediction and experimental measurement.
On a related note, the difference between model and experiment can also be reduced by
choosing a closed, instead of an open, model. is is an issue that is examined in OpenClosed [35].
9.2.3 REAL SYSTEMS AND WORKLOADS
Ideally, a model should be validated with the real system itself— for example, the experiments in
InternetServices [48] used actual implementations of the servers and applications. Unfortunately,
this is not always possible. Database administrators loathe experiments with production systems,
and vendors zealously guard against publication of performance measurements of their products.
One possible compromise is to log the events in a real system, then replay them on a simu-
lator. Replaying such a trace would be more realistic than running a synthetic workload (like the
TPC-C benchmark used in SleepingDisks [51]). One could even massage the trace by running it
at a different speed, or cutting it and playing different pieces concurrently, etc.
However, the realism is also a weakness. For example, the trace from a database system is
already filtered through its concurrency control, possibly altering the spatial and temporal corre-
lation in the trace. One may therefore get misleading results from playing it on another system,
with a different concurrency control, or at a different multiprogramming level.

Get Analytical Performance Modeling for Computer Systems, 2nd Edition now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.