Design, Evaluation, and Analysis of Questionnaires for Survey Research, 2nd Edition

10

ESTIMATION OF RELIABILITY, VALIDITY, AND METHOD EFFECTS

In the last chapter, we discussed several criteria to consider while thinking about the quality of survey measures. Two criteria, item nonresponse and bias, do not require much further discussion. If different methods have been used for the same trait, item nonresponse can be observed directly from collected data. The same holds true for bias if factual information is available. However, this is not the case if an estimate of the relative bias of measures can be derived by comparing distributions of responses with each other. Molenaar (1986) has made useful suggestions for measures of relative bias.

More complicated is the estimation of the quality criteria of reliability, validity, and method effect. Therefore, this chapter and the next will concentrate on their estimation. In order to discuss the estimation of reliability and validity, we have to introduce the basic idea behind estimation of coefficients of measurement models. So far, it is not so evident that the effect of an unmeasured variable on a measured variable, let alone the effect of an unmeasured variable on another unmeasured variable, can be estimated.

Here, we start by addressing the problem of identifying the parameters of models with unmeasured variables. Next, we will discuss the estimation of the parameters. Only after we have introduced these basic principles we will concentrate on the estimation of the reliability and validity, demonstrating the kind ...