Audio Signal Processing and Coding

12.3 CONFOUNDING FACTORS IN SUBJECTIVE EVALUATIONS

Regardless of the particular grading scale in use, subjective test outcomes generated using even rigorous methodologies such as the ITU-R BS.1116 are still influenced by factors such as context, site selection, and individual listener acuity (physical) or preference (cognitive). Before comparing subjective test results on particular codecs, therefore, one should be prepared to interpret the subjective scores with some care. For example, consider the variability of “expert” listeners. A study of decision strategies [Prec97] using multidimensional scaling techniques [Schi81] found that subjects disagree on the relative importance with which to weigh perceptual criteria during impairment detection tasks. In another study [Shli96], Shlien and Soulodre presented experimental evidence that can be interpreted as a repudiation of the “golden ear.” Expert listeners were tasked with discriminating between clean audio and audio corrupted by low-level artifacts typically induced by audio codecs (five types were analyzed in [Miln92]), including pre-echo distortion, unmasked granular (quantization) noise, and high-frequency boost or attenuation. Different experts were sensitive to different artifact types. For instance, a subject strongly sensitive to pre-echo distortion was not necessarily proficient when asked to detect high-frequency gain or attenuation. In fact, artifact sensitivities were linked to psychophysical measurement profiles. Because ...