A PREDICTIVE METHOD TO MEASURE
RELATIVE EFFECTIVENESS
Nan Jiang
Software Research Centre, Bournemouth University, UK
Errors and completion are two widely used metrics for measuring the
effectiveness of a system in usability testing. Since the two measures
focus on different aspects of user output, a holistic view of “effec-
tiveness” is sometimes hard to establish in a comparative study which
eventually affects understanding different systems’ relative effec-
tiveness. The paper proposes a predictive method using an adapted
confusion matrix to establish a correlation model to measure a sys-
tem’s relative effectiveness based on its own performance prediction.
A case study is also provided to demonstrate how to use this method
in real-world practice.
Introduction
ISO 9244-11:1998 defines effectiveness as the accuracy and completeness with
which users achieve specified goals. According to ISO/IEC 25062:2006 CIF, the
two factors are often reflected by errors and completion respectively in usability
testing. In practice, considering the complex causes of human errors in task per-
formance (Newell et al, 1958) (Rasmussen & Vicente, 1989), a system will be
generally considered as effective if users made few errors and achieved high task
completion rate on the system in a performance test. The guidance seems fine to
be used for understanding a system’s effectiveness independently or comparing the
system before and after improvements. However, this holistic view may not be eas-
ily applied to a comparative study to understand a system’s relative effectiveness for
two reasons. First, a correlation between errors and completion is complex to model
as they reflect different aspects of user output. Second, as Coll and Wingertsman
(1990) stated, when systems show differences in magnitude and complexity, not
only can user’s performance on these systems be affected by such variables, their
impact on effectiveness is also hard to measure.
This paper proposes a predictive method to measure a system’s relative effec-
tiveness based on the general grounds of an ‘effective’ system described above.
This is achieved through using an adapted confusion matrix to present a practical
correlation model between the two metrics and apply standard confusion matrix
measurements to predict a system’s relative effectiveness in a self-benchmarking
process. The main benefits are obvious. First, it provides a holistic view of effec-
tiveness with its two common measures. Second, it establishes a simplified measure
for assessing a system’s relative effectiveness by predicting at which error margin
http://dx.doi.org/10.1201/b13826-14
43

Get Contemporary Ergonomics and Human Factors 2013 now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.