Item Response Theory

8Computerized Adaptive Testing

In our lust for measurement, we frequently measure that which we can rather than that which we wish to measure and forget that there is a difference.

(Source: George Udny Yule)

8.1 What Is Computerized Adaptive Testing?

Imagine a 1000‐item mathematics test with items ranging in difficulty from basic arithmetic through advanced calculus. Now consider two examinees, a fourth‐grader and a graduate student in mathematics. Most questions will be uninformative for both examinees (too difficult for the first and too easy for the second). To decrease examinee burden, we could create a short test of 10 items, equally spaced along the mathematics difficulty continuum. Although this test would be quick to administer, it would provide very imprecise estimates of their abilities because only an item or two would be appropriate for either examinee. A better approach would be to begin by administering an item of intermediate difficulty, and based on the response scored as correct or incorrect, select the next item at a level of difficulty either lower or higher. This process would continue until the uncertainty in the estimated ability is smaller than a predefined threshold. This process is called computerized adaptive testing (CAT). To use CAT, we must first calibrate a bank of test items using an item response theory (IRT) model that relates properties of ...