Developing Data Migrations and Integrations with Salesforce: Patterns and Best Practices

Appendix A: A Simple Duplicate Detection Algorithm

I prepared this appendix for you because data cleansing and duplicate management is such an important part of a data migration or integration specialist’s job. ¹

The two primary reasons companies migrate off one system to another is the lack of needed functionality and missing controls to maintain data quality properly. In general, users are very resourceful and can find clever ways to overcome system limitations . They use fields or features to do things for which they were not intended, simply so they can do their job. Over time, the system’s data becomes messy and inaccurate, leading to loss of system usefulness.

If we were to migrate data from such a system as is, we run ...