97 Things Every Data Engineer Should Know

Chapter 75. The Many Meanings of Missingness

Emily Riederer

Missing data is possibly one of the best-explored topics in traditional data management; preventing null values is the textbook example for a database constraint, and detecting nulls is the same for data validation. However, missing data should not simply be dodged. By considering the many meanings of missing, data engineers can make more-thoughtful decisions about how to encode and communicate this missingness to end users.

It’s tempting to think of missingness as a binary: the data exists or does not. However, this view ignores the potentially informative nature of nulls. Missingness can arise in numerous ways, each with its own analytical challenges. For example, a variable for an observation might appear to be missing because of the following:

• A true value for that variable exists, but it was loaded incorrectly.

• A true value for that variable exists, but it was intentionally uncollected.

• A true value for that variable exists, but it is unknown.

• No relevant value exists for that variable.

• The relevant value for that variable is null.

To illustrate, consider a dataset of users registering for an online ecommerce platform:

• A mismatch in account ID formats across tables, or arithmetic performed using a null and a non-null field, could introduce null values.

• A random sample of users is invited to complete a survey, ...