Statistics of Medical Imaging

12

Performance Ev aluation of Image Analysis

Methods

12.1 Introduction

With the rapid development of image analysis techniques [1 –39], an increasing

interest has been directed toward the performance evaluation of these tech-

niques. Commonly used evaluation criteria may include accuracy, precisio n,

efficiency, consistency, reproducibility, r obustness, etc. In orde r to make assess-

ments based on these criteria (e.g., the accura cy), observers often compare the

results obtained using these techniques with the corres ponding ground tr uth

or the gold standard.

The ground truth may be seen as a conceptual term relative to the knowl-

edge of the truth concerning a specific question. The gold standard may be

seen as the concrete realizatio n of the g round truth or an accepted sur rogate

of truth [40 ]. Due to the complexity of the structures of living objects a nd

the irregularity of the anomalie s, the ground truth or the gold standard of

these structures and a nomalies is unknown, inaccurate, or even difficult to

establish. As a result, subjective criteria and procedur es are often used in the

performance evaluation, which can le ad to inac curate or biased asses sments.

This chapter describes two approaches for the precise and quantitative eval-

uation of the performa nc e of imag e ana lysis techniques. Instead of comparing

the re sults obtained by the image analysis techniques with the ground truth or

the gold standard, or using some statistical measures, these two approaches

directly assess the image analysis technique itself. The first approach gives

analytical assessments of the performance of each step of the image analy-

sis technique. The second appr oach is focused on the validity of the image

analysis technique with its fundamental imaging principles.

The first approach is applied to the iFNM model-bas ed image analysis

metho d (Chapter 10), which consists of three steps: detection, estimation, and

classification. (1) For detection performance, pro babilities of over- and under-

detection of the number of image regions are defined, and the corresponding

formulas in terms of model parameters and image quality are derived. (2) For

estimation performance, both EM and CM algorithms are showed to produce

asymptotically unbiased ML estimates of model parameters in the case of

no-overlap. Cramer-Rao bounds of the varia nc es of these estimates are de-

371

372 Statistics of Medical Imaging

rived. (3) For classification pe rformance, a misclassification probability for

the Bayesian clas sifier is defined, and a simple formula, based on parameter

estimates a nd classified data, is derived to e valuate classification errors.

The results obtained by applying this method to a set of simulated im-

ages show that, for images with a moderate quality (SN R > 14.2 db, i.e.,

µ/σ ≥ 5.13), (1) the number of image regions suggested by the detection

criterion is correct and the error-detection probabilities are almo st zero; (2)

the relative errors of the weight and the mean are less than 0.6%, and all

parameter estimates are in the C ramer-Rao estimation intervals; and (3) the

misclassification probabilities are le ss than 0.5%. These results demonstrate

that for this class of image ana lysis methods, the detection procedure is ro-

bust, the parameter estimates ar e a ccurate, and the classificatio n errors a re

small.

A strength of this approach is that it not only provides the theoretically

approachable accuracy limits of image analysis techniques, but also shows the

practically a chievable performance for the given image s.

The second appr oach is applied to the cFNM model-based image analys is

metho d (Chapter 11), which also consists of three steps: detection, estima-

tion, and classification. (1) For detection per formance, although the cFNM

model-based image ana lysis method use s a sensor ar ray eigenstructure-based

approach (which is different from the information criterion-based approach

used in the iFNM model-based image analysis method), the probabilities of

over- and under-detection of the number of image regions are defined in a sim-

ilar way. The error-detection probabilities are shown to be functions of image

quality, resolution, and complexity. (2) For estimation performance, when the

EM algorithm is used, the performances of iFNM and cFNM model-based

image analys is methods are similar. (3) For classification p erformance, the

cFNM model-based image analysis method uses the MAP criterion to assess

its validity with the underlying imaging principles and shows that the results

obtained by MAP are toward the physical ground truth that is to be imaged.

12.2 Performance of the iFNM Model-Based Image

Analysis Method

This section analyzes the iFNM model-based image analysis method of Cha p-

ter 10. It evaluates its performance at three steps: detection, estimation, and

classification.