Learning Data Mining with Python - Second Edition

Ranking mechanisms for word similarity

The Levenshtein edit distance is a commonly used method for comparing two short strings to see how similar they are. It isn't very scalable, so it isn't commonly used for very long strings. The edit distance computes the number of steps it takes to go from one word to another. The steps can be one of the following three actions:

• Insert a new letter into the word at any position
• Delete any letter from the word
• Substitute a letter for another one

The minimum number of actions needed to transform the first word into the second is given as the distance. Higher values indicate that the words are less similar.

This distance is available in NLTK as nltk.metrics.edit_distance. We can call it using only two ...