Selective Visual Attention: Computational Models and Applications

6.4 Quantitative Evaluation

As mentioned above, the ground-truth data can be obtained from man-made visual patterns, human labelling and eye-tracking. Using these data one can evaluate the performance of a visual attention model in two ways: qualitative evaluation and quantitative evaluation. Qualitative evaluation involves the comparison of the computed saliency maps with the ground-truth data by visual inspection. However, this is a rather crude method and cannot be used in real-time scenarios. Moreover, it may give less consistent results since it is based on manual inspection. In order to overcome these limitations, quantitative evaluation can be used for a more accurate comparison of visual attention models. To that end, three commonly used criteria are used: one is related to classification such as precision estimation or F-measure, the receiver operating characteristic curve (ROC curve) or area under the ROC curve (AUC) score and so on. The second one is based on Bayesian surprise (discussed in Section 3.8) and is called the KL score, and the final estimated method is Spearman's rank order correlation that is to be introduced in Section 6.6. This section mainly introduces the estimation related to classification.

First, the ground-truth data are converted into binary saliency maps: a value of 1 indicates fixation locations and 0 indicates non-fixation locations. Man-made visual patterns and human-labelling data are easy to derive the binary format (as Figure 6.1(a) and ...