Hands-On Entity Resolution

Chapter 5. Record Blocking

In Chapter 4, we introduced probabilistic matching techniques to allow us to combine exact equivalence on individual attributes into a weighted composite score. That score allowed us to calculate the overall probability that two records refer to the same entity.

So far we have sought to resolve only small-scale datasets where we could exhaustively compare every record with every other to find all possible matches. However, in most entity resolution scenarios, we will be dealing with larger datasets where this approach isn’t practical or affordable.

In this chapter we will introduce record blocking to reduce the number of permutations we need to consider while minimizing the likelihood of missing a true positive match. We will leverage the Splink framework, introduced in the last chapter, to apply the Fellegi-Sunter model and use the expectation-maximization algorithm to estimate the model parameters.

Lastly, we will consider how to measure our matching performance over this larger dataset.

Sample Problem

In previous chapters, we considered the challenge of resolving entities across two datasets containing information about members of the UK House of Commons. In this chapter, we extend this resolution challenge to a much larger dataset containing a list of the persons with significant control of registered UK companies.

In the UK, Companies House is an executive agency sponsored by the Department for Business and Trade. It incorporates and dissolves ...