Total Survey Error in Practice

22Analytic Error as an Important Component of Total Survey Error: Results from a Meta‐Analysis

Brady T. West,^1,2 Joseph W. Sakshaug,^3,4 and Yumi Kim⁵

¹ Survey Research Center, University of Michigan, Ann Arbor, MI, USA

² Joint Program in Survey Methodology, University of Maryland, College Park, MD, USA

³ Cathie Marsh Institute for Social Research, University of Manchester, Manchester, UK

⁴ Department of Statistical Methods, Institute for Employment Research (IAB), Nuremberg, Germany

⁵ Department of Research Methods, Market Strategies International, Livonia, MI, USA

22.1 Overview

We describe the results of a meta‐analysis that sought to quantify the frequency with which researchers do not use correct analytic techniques for complex sample survey data when publishing secondary analyses of survey data arising from complex samples. The analytic errors that may result from incorrect analyses of these data represent an important but understudied aspect of the total survey error (TSE) paradigm, and can negate the positive data quality aspects introduced by dedicating substantial resources to the minimization of other important sources of survey error (e.g., nonresponse error and measurement error). We also sought to explore whether characteristics of the journals in which these articles were published (e.g., impact factor, presence of statisticians on the editorial boards, analytic guidelines for authors, etc.) were related to the prevalence of various errors. The results of the ...