Design, Evaluation, and Analysis of Questionnaires for Survey Research, 2nd Edition

12

MTMM EXPERIMENTS AND THE QUALITY OF SURVEY QUESTIONS

In Chapter 10, the classical MTMM design suggested by Campbell and Fiske (1959) and the analysis of the data obtained by such an experiment using the true score MTMM model have been discussed. This approach has been used in many experiments in the period between 1979 and 1997 by Andrews (1984) and Rodgers et al. (1992) in the United States, Költringer (1995) in Austria, Scherpenzeel and Saris (1997) in the Netherlands, and Billiet and Waege (1989, 1997) in Flanders (Belgium). Scherpenzeel and Saris also used frequently a variant of this design using a panel. In Chapter 11, we have described the SB-MTMM design and analysis developed by Saris et al. (2004). This approach has been used in the European Social Survey (ESS) in the period between 2001 and 2012.

In the earlier studies, the emphasis was much more on variation in the form of the questions, while in the ESS, the emphasis was more on the comparability of the questions across countries and the differences due to the use of different languages and cultures.

In this chapter, an overview is given on the MTMM experiments that have been done from 1979 till 2007. Close to 4000 questions have been evaluated in this way. This research has delivered a lot of information. We will provide some information about these questions. However, in order to correct for measurement error in survey research the obtained information is very little. At this moment, already more than 60,000 questions ...