Total Survey Error in Practice

Total Survey Error in Practice

by Paul P. Biemer, Edith de Leeuw, Stephanie Eckman, Brad Edwards, Frauke Kreuter, Lars E. Lyberg, N. Clyde Tucker, Brady T. West
Released February 2017
Publisher(s): Wiley
ISBN: 9781119041672

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Featuring a timely presentation of total survey error (TSE), this edited volume introduces valuable tools for understanding and improving survey data quality in the context of evolving large-scale data sets

This book provides an overview of the TSE framework and current TSE research as related to survey design, data collection, estimation, and analysis. It recognizes that survey data affects many public policy and business decisions and thus focuses on the framework for understanding and improving survey data quality. The book also addresses issues with data quality in official statistics and in social, opinion, and market research as these fields continue to evolve, leading to larger and messier data sets. This perspective challenges survey organizations to find ways to collect and process data more efficiently without sacrificing quality. The volume consists of the most up-to-date research and reporting from over 70 contributors representing the best academics and researchers from a range of fields. The chapters are broken out into five main sections: The Concept of TSE and the TSE Paradigm, Implications for Survey Design, Data Collection and Data Processing Applications, Evaluation and Improvement, and Estimation and Analysis. Each chapter introduces and examines multiple error sources, such as sampling error, measurement error, and nonresponse error, which often offer the greatest risks to data quality, while also encouraging readers not to lose sight of the less commonly studied error sources, such as coverage error, processing error, and specification error. The book also notes the relationships between errors and the ways in which efforts to reduce one type can increase another, resulting in an estimate with larger total error.

This book:

• Features various error sources, and the complex relationships between them, in 25 high-quality chapters on the most up-to-date research in the field of TSE

• Provides comprehensive reviews of the literature on error sources as well as data collection approaches and estimation methods to reduce their effects

• Presents examples of recent international events that demonstrate the effects of data error, the importance of survey data quality, and the real-world issues that arise from these errors

• Spans the four pillars of the total survey error paradigm (design, data collection, evaluation and analysis) to address key data quality issues in official statistics and survey research

Total Survey Error in Practice is a reference for survey researchers and data scientists in research areas that include social science, public opinion, public policy, and business. It can also be used as a textbook or supplementary material for a graduate-level course in survey research methods.

Paul P. Biemer, PhD, is distinguished fellow at RTI International and associate director of Survey Research and Development at the Odum Institute, University of North Carolina, USA.

Edith de Leeuw, PhD, is professor of survey methodology in the Department of Methodology and Statistics at Utrecht University, the Netherlands.

Stephanie Eckman, PhD, is fellow at RTI International, USA.

Brad Edwards is vice president, director of Field Services, and deputy area director at Westat, USA.

Frauke Kreuter, PhD, is professor and director of the Joint Program in Survey Methodology, University of Maryland, USA; professor of statistics and methodology at the University of Mannheim, Germany; and head of the Statistical Methods Research Department at the Institute for Employment Research, Germany.

Lars E. Lyberg, PhD, is senior advisor at Inizio, Sweden.

N. Clyde Tucker, PhD, is principal survey methodologist at the American Institutes for Research, USA.

Brady T. West, PhD, is research associate professor in the Survey Resea

Table of contents

  1. Cover
  2. Title Page
  3. Notes on Contributors
  4. Preface
  5. Section 1: The Concept of TSE and the TSE Paradigm
    1. 1 The Roots and Evolution of the Total Survey Error Concept
      1. 1.1 Introduction and Historical Backdrop
      2. 1.2 Specific Error Sources and Their Control or Evaluation
      3. 1.3 Survey Models and Total Survey Design
      4. 1.4 The Advent of More Systematic Approaches Toward Survey Quality
      5. 1.5 What the Future Will Bring
      6. References
    2. 2 Total Twitter Error
      1. 2.1 Introduction
      2. 2.2 Social Media: An Evolving Online Public Sphere
      3. 2.3 Components of Twitter Error
      4. 2.4 Studying Public Opinion on the Twittersphere and the Potential Error Sources of Twitter Data: Two Case Studies
      5. 2.5 Discussion
      6. 2.6 Conclusion
      7. References
    3. 3 Big Data
      1. 3.1 Introduction
      2. 3.2 Definitions
      3. 3.3 The Analytic Challenge: From Database Marketing to Big Data and Data Science
      4. 3.4 Assessing Data Quality
      5. 3.5 Applications in Market, Opinion, and Social Research
      6. 3.6 The Ethics of Research Using Big Data
      7. 3.7 The Future of Surveys in a Data‐Rich Environment
      8. References
    4. 4 The Role of Statistical Disclosure Limitation in Total Survey Error
      1. 4.1 Introduction
      2. 4.2 Primer on SDL
      3. 4.3 TSE‐Aware SDL
      4. 4.4 Edit‐Respecting SDL
      5. 4.5 SDL‐Aware TSE
      6. 4.6 Full Unification of Edit, Imputation, and SDL
      7. 4.7 “Big Data” Issues
      8. 4.8 Conclusion
      9. Acknowledgments
      10. References
  6. Section 2: Implications for Survey Design
    1. 5 The Undercoverage–Nonresponse Tradeoff
      1. 5.1 Introduction
      2. 5.2 Examples of the Tradeoff
      3. 5.3 Simple Demonstration of the Tradeoff
      4. 5.4 Coverage and Response Propensities and Bias
      5. 5.5 Simulation Study of Rates and Bias
      6. 5.6 Costs
      7. 5.7 Lessons for Survey Practice
      8. References
    2. 6 Mixing Modes
      1. 6.1 Introduction
      2. 6.2 The Effect of Offering a Choice of Modes
      3. 6.3 Getting People to Respond Online
      4. 6.4 Sequencing Different Modes of Data Collection
      5. 6.5 Separating the Effects of Mode on Selection and Reporting
      6. 6.6 Maximizing Comparability Versus Minimizing Error
      7. 6.7 Conclusions
      8. References
    3. 7 Mobile Web Surveys
      1. 7.1 Introduction
      2. 7.2 Coverage
      3. 7.3 Nonresponse
      4. 7.4 Measurement Error
      5. 7.5 Links Between Different Error Sources
      6. 7.6 The Future of Mobile web Surveys
      7. References
    4. 8 The Effects of a Mid‐Data Collection Change in Financial Incentives on Total Survey Error in the National Survey of Family Growth
      1. 8.1 Introduction
      2. 8.2 Literature Review: Incentives in Face‐to‐Face Surveys
      3. 8.3 Data and Methods
      4. 8.4 Results
      5. 8.5 Conclusion
      6. References
    5. 9 A Total Survey Error Perspective on Surveys in Multinational, Multiregional, and Multicultural Contexts
      1. 9.1 Introduction
      2. 9.2 TSE in Multinational, Multiregional, and Multicultural Surveys
      3. 9.3 Challenges Related to Representation and Measurement Error Components in Comparative Surveys
      4. 9.4 QA and QC in 3MC Surveys
      5. References
    6. 10 Smartphone Participation in Web Surveys
      1. 10.1 Introduction
      2. 10.2 Prevalence of Smartphone Participation in Web Surveys
      3. 10.3 Smartphone Participation Choices
      4. 10.4 Instrument Design Choices
      5. 10.5 Device and Design Treatment Choices
      6. 10.6 Conclusion
      7. 10.7 Future Challenges and Research Needs
      8. Appendix 10.A: Data Sources
      9. Appendix 10.B: Smartphone Prevalence in Web Surveys
      10. Appendix 10.C: Screen Captures from Peterson et al. (2013) Experiment
      11. Appendix 10.D: Survey Questions Used in the Analysis of the Peterson et al. (2013) Experiment
      12. References
    7. 11 Survey Research and the Quality of Survey Data Among Ethnic Minorities
      1. 11.1 Introduction
      2. 11.2 On the Use of the Terms Ethnicity and Ethnic Minorities
      3. 11.3 On the Representation of Ethnic Minorities in Surveys
      4. 11.4 Measurement Issues
      5. 11.5 Comparability, Timeliness, and Cost Concerns
      6. 11.6 Conclusion
      7. References
  7. Section 3: Data Collection and Data Processing Applications
    1. 12 Measurement Error in Survey Operations Management
      1. 12.1 TSE Background on Survey Operations
      2. 12.2 Better and Better: Using Behavior Coding (CARIcode) and Paradata to Evaluate and Improve Question (Specification) Error and Interviewer Error
      3. 12.3 Field‐Centered Design: Mobile App for Rapid Reporting and Management
      4. 12.4 Faster and Cheaper: Detecting Falsification With GIS Tools
      5. 12.5 Putting It All Together: Field Supervisor Dashboards
      6. 12.6 Discussion
      7. References
    2. 13 Total Survey Error for Longitudinal Surveys
      1. 13.1 Introduction
      2. 13.2 Distinctive Aspects of Longitudinal Surveys
      3. 13.3 TSE Components in Longitudinal Surveys
      4. 13.4 Design of Longitudinal Surveys from a TSE Perspective
      5. 13.5 Examples of Tradeoffs in Three Longitudinal Surveys
      6. 13.6 Discussion
      7. References
    3. 14 Text Interviews on Mobile Devices
      1. 14.1 Texting as a Way of Interacting
      2. 14.2 Contacting and Inviting Potential Respondents through Text
      3. 14.3 Texting as an Interview Mode
      4. 14.4 Costs and Efficiency of Text Interviewing
      5. 14.5 Discussion
      6. References
    4. 15 Quantifying Measurement Errors in Partially Edited Business Survey Data
      1. 15.1 Introduction
      2. 15.2 Selective Editing
      3. 15.3 Effects of Errors Remaining After SE
      4. 15.4 Case Study: Foreign Trade in Goods Within the European Union
      5. 15.5 Editing Big Data
      6. 15.6 Conclusions
      7. References
  8. Section 4: Evaluation and Improvement
    1. 16 Estimating Error Rates in an Administrative Register and Survey Questions Using a Latent Class Model
      1. 16.1 Introduction
      2. 16.2 Administrative and Survey Measures of Neighborhood
      3. 16.3 A Latent Class Model for Neighborhood of Residence
      4. 16.4 Results
      5. 16.5 Discussion and Conclusion
      6. Appendix 16.A: Program Input and Data
      7. Acknowledgments
      8. References
    2. 17 ASPIRE
      1. 17.1 Introduction and Background
      2. 17.2 Overview of ASPIRE
      3. 17.3 The ASPIRE Model
      4. 17.4 Evaluation of Registers
      5. 17.5 National Accounts
      6. 17.6 A Sensitivity Analysis of GDP Error Sources
      7. 17.7 Concluding Remarks
      8. Appendix 17.A: Accuracy Dimension Checklist
      9. References
    3. 18 Classification Error in Crime Victimization Surveys
      1. 18.1 Introduction
      2. 18.2 Background
      3. 18.3 Analytic Approach
      4. 18.4 Model Selection
      5. 18.5 Results
      6. 18.6 Discussion and Summary of Findings
      7. 18.7 Conclusions
      8. Appendix 18.A: Derivation of the Composite False‐Negative Rate
      9. Appendix 18.B: Derivation of the Lower Bound for False‐Negative Rates from a Composite Measure
      10. Appendix 18.C: Examples of Latent GOLD Syntax
      11. References
    4. 19 Using Doorstep Concerns Data to Evaluate and Correct for Nonresponse Error in a Longitudinal Survey
      1. 19.1 Introduction
      2. 19.2 Data and Methods
      3. 19.3 Results
      4. 19.4 Discussion
      5. Acknowledgment
      6. References
    5. 20 Total Survey Error Assessment for Sociodemographic Subgroups in the 2012 U.S. National Immunization Survey
      1. 20.1 Introduction
      2. 20.2 TSE Model Framework
      3. 20.3 Overview of the National Immunization Survey
      4. 20.4 National Immunization Survey: Inputs for TSE Model
      5. 20.5 National Immunization Survey TSE Analysis
      6. 20.6 Summary
      7. References
    6. 21 Establishing Infrastructure for the Use of Big Data to Understand Total Survey Error
      1. Overview
      2. Part 1 Big Data Infrastructure at the Institute for Employment Research (IAB)
      3. References
      4. Part 2 Using Administrative Records Data at the U.S. Census Bureau: Lessons Learned from Two Research Projects Evaluating Survey Data
      5. References
      6. Part 3 Statistics New Zealand’s Approach to Making Use of Alternative Data Sources in a New Era of Integrated Data
      7. References
      8. Part 4 Big Data Serving Survey Research: Experiences at the University of Michigan Survey Research Center
      9. References
  9. Section 5: Estimation and Analysis
    1. 22 Analytic Error as an Important Component of Total Survey Error
      1. 22.1 Overview
      2. 22.2 Analytic Error as a Component of TSE
      3. 22.3 Appropriate Analytic Methods for Survey Data
      4. 22.4 Methods
      5. 22.5 Results
      6. 22.6 Discussion
      7. Acknowledgments
      8. References
    2. 23 Mixed‐Mode Research
      1. 23.1 Introduction
      2. 23.2 Designing Mixed‐Mode Surveys
      3. 23.3 Literature Overview
      4. 23.4 Diagnosing Sources of Error in Mixed‐Mode Surveys
      5. 23.5 Adjusting for Mode Measurement Effects
      6. 23.6 Conclusion
      7. References
    3. 24 The Effect of Nonresponse and Measurement Error on Wage Regression across Survey Modes
      1. 24.1 Introduction
      2. 24.2 Nonresponse and Response Bias in Survey Statistics
      3. 24.3 Data and Methods
      4. 24.4 Results
      5. 24.5 Summary and Conclusion
      6. Acknowledgments
      7. Appendix 24.A
      8. Appendix 24.B
      9. References
    4. 25 Errors in Linking Survey and Administrative Data
      1. 25.1 Introduction
      2. 25.2 Conceptual Framework of Linkage and Error Sources
      3. 25.3 Errors Due to Linkage Consent
      4. 25.4 Erroneous Linkage with Unique Identifiers
      5. 25.5 Erroneous Linkage with Nonunique Identifiers
      6. 25.6 Applications and Practical Guidance
      7. 25.7 Conclusions and Take‐Home Points
      8. References
  10. Wiley Series in Survey Methodology
  11. Index
  12. End User License Agreement

Product information

  • Title: Total Survey Error in Practice
  • Author(s): Paul P. Biemer, Edith de Leeuw, Stephanie Eckman, Brad Edwards, Frauke Kreuter, Lars E. Lyberg, N. Clyde Tucker, Brady T. West
  • Release date: February 2017
  • Publisher(s): Wiley
  • ISBN: 9781119041672