Big Data Meets Survey Science

14Using Machine Learning Models to Predict Attrition in a Survey Panel

Mingnan Liu

Independent Researcher, San Mateo, CA, USA

14.1 Introduction

A longitudinal or panel survey differs from a cross‐sectional survey in that it repeatedly collects data from the same group of respondents to monitor changes of attitudes, opinions, or behaviors over time. An alternative approach for measuring changes is to ask retrospective questions in a cross‐sectional survey. That approach, however, is subject to recall error, so in many cases longitudinal or panel surveys are preferable to facilitate understanding changes and trends (Olsen 2018).

Although obtaining high or adequate response rates is increasingly challenging for any survey, longitudinal or panel surveys face the additional challenge of panel attrition – that is, respondents who participate in one interview but not subsequent interviews. Researchers who conduct these types of surveys not only have to recruit participants at the first stage but they must also essentially re‐recruit these original respondents to participate in further surveys at later stages.

Lynn (2018) summarizes two main reasons why panel attrition is problematic. First, a high panel attrition rate can cause the sample size of subsequent waves of surveys to shrink rapidly, which hinders or prevents follow‐up surveys to produce precise estimates for the survey population. Second, similar to survey nonresponse bias, panel attrition can introduce bias to survey estimates, ...