97 Things Every Data Engineer Should Know

Chapter 59. QA and All Its Sexiness

Sonia Mehta

Before moving into a new home, prospective homeowners will hire an inspector to assess any damage within the house. Much like house inspectors, as data engineers, it’s on us to spot any glaring and not-so-glaring issues with our data before sending it to production.

Setting up a quality assurance (QA) program is simple, and the gains are totally worth it! In setting up a QA program, tests can be divided into two major buckets: practical and logical.

Practical tests aim to test for completeness of the data and accurate data types. These include the following:

• Checking for data coverage by inspecting dates or expected row counts

• Standardizing currency inputs (e.g., removing commas from metrics)

• Ensuring that required fields contain no null values

• Validating for consistent date, time zone, and casing formats

• Confirming that headers are applied to data and not within the data itself

• Deduping the dataset

Logical tests are for business and domain relevance. This is the fun part! Integral to this step is obtaining a macro business context and understanding the main questions looking to be answered. It’s also helpful to understand how important accuracy is to the stakeholder. Is directional accuracy sufficient, or is complete accuracy required? (Oftentimes finance teams will say that total accuracy is required, whereas other teams may ...