Fuzzy Data Matching with SQL

Glossary

customer dataset

Also known as production data, as CRM data, and by other similar terms in this book. This is the production dataset you are going to be comparing the incoming data against for matches.

data quality

Data represented in a way that minimizes error while maximizing utility. This includes having consistency in encoding, abbreviations, spelling, punctuation, and more.

dataset

A “rectangular” collection of one or more rows (i.e., with a fixed number of columns). Each column is named and optionally typed. A dataset can be stored and accessed via a SQL table, comma-separated value (CSV) or tab-separated value (TSV), Excel file, XML, JSON, or whatever.

EDA

“Exploratory data analysis.” This is a fancy term for running some really simple queries and statistics against a dataset to see what’s there. I often informally call it “sniffing the data” to look for patterns, errors, data distributions, and the like.

ETL

“Extract, transform, and load.” That is, extract data from somewhere (table, flat file, wherever), transform it by applying various filters and alterations to the data, and load the results into a new location, which in the SQL world is typically a(nother) table.

fuzzy

Attempting to match between heterogeneous datasets that represent the “same data” in differing data entry formats, schemas, datatypes, encodings, conventions, and locales—and sometimes even different semantics (see Chapter 9).

import dataset

Also known as “incoming data.” This is the dataset ...