Automating Data Quality Monitoring

Appendix. Types of Data Quality Issues

This appendix presents additional information about the types of data quality issues that are commonly encountered in real-world data. This list is helpful to consider as you evaluate the data quality monitoring solution you are building or buying. Ultimately, you’ll want to have a strategy for identifying and addressing each of these types of issues for each important dataset in your organization.

For each of these data quality issues, we will provide an example, a summary of common causes, an assessment of how these issues typically affect analytics (using data and humans to inform decisions) and machine learning (using data and algorithms to automate processes), and our recommendations for how best to monitor a data source for these issues.

Figure A-1. Types of data quality issues organized into four categories (DALL-E 3)

As Figure A-1 shows, we have organized the issues in this appendix into four broad categories that indicate at what level the issues affect data.

• Table issues

  Issues that affect the entirety of the table, and aren’t specific to individual rows or values:

  Late arrival

  When data arrives late and is not available to a consuming system by the time the system needs the data

  Schema changes

  When there are structural changes in the data such as new or dropped columns, changes in column names, changes in data types for columns, ...