78 9. EXPERIMENTAL VALIDATION AND ANALYSIS
In the case of database locking, for example, should a transaction’s own locks be taken into
account in calculating the probability of a conﬂict? Not having to do so may greatly simplify the
analysis. One could make the choice by developing the model both ways, and using simulation
measurements to see if the simpliﬁcation causes a signiﬁcant loss in accuracy.
9.2.2 DATA PRESENTATION
In presenting experimental data to validate a model, one should bear in mind the approximations
in the model and the accuracy of the experiment. For example, it is not meaningful to use multiple
decimal places when such precision is unwarranted.
It is often easier to compare numerical values with a data plot. However, one’s judgment
can be skewed if the axes are translated or scaled. For instance, a vertical axis that does not start
at 0 can exaggerate the performance diﬀerence between two algorithms, and a logarithmic axis
can understate the diﬀerence between model prediction and experimental measurement.
On a related note, the diﬀerence between model and experiment can also be reduced by
choosing a closed, instead of an open, model. is is an issue that is examined in OpenClosed .
9.2.3 REAL SYSTEMS AND WORKLOADS
Ideally, a model should be validated with the real system itself— for example, the experiments in
InternetServices  used actual implementations of the servers and applications. Unfortunately,
this is not always possible. Database administrators loathe experiments with production systems,
and vendors zealously guard against publication of performance measurements of their products.
One possible compromise is to log the events in a real system, then replay them on a simu-
lator. Replaying such a trace would be more realistic than running a synthetic workload (like the
TPC-C benchmark used in SleepingDisks ). One could even massage the trace by running it
at a diﬀerent speed, or cutting it and playing diﬀerent pieces concurrently, etc.
However, the realism is also a weakness. For example, the trace from a database system is
already ﬁltered through its concurrency control, possibly altering the spatial and temporal corre-
lation in the trace. One may therefore get misleading results from playing it on another system,
with a diﬀerent concurrency control, or at a diﬀerent multiprogramming level.