Chapter 21. Reducing Words to Their Root Form
Most languages of the world are inflected, meaning that words can change their form to express differences in the following:
Number: fox, foxes
Tense: pay, paid, paying
Gender: waiter, waitress
Person: hear, hears
Case: I, me, my
Aspect: ate, eaten
Mood: so be it, were it so
While inflection aids expressivity, it interferes with retrievability, as a single root word sense (or meaning) may be represented by many different sequences of letters. English is a weakly inflected language (you could ignore inflections and still get reasonable search results), but some other languages are highly inflected and need extra work in order to achieve high-quality search results.
Stemming attempts to remove the differences between inflected forms of a
word, in order to reduce each word to its root form. For instance
be reduced to the root
fox, to remove the difference between singular and
plural in the same way that we removed the difference between lowercase and
The root form of a word may not even be a real word. The words
jumpiness may both be stemmed to
jumpi. It doesn’t matter—as long as
the same terms are produced at index time and at search time, search will just
If stemming were easy, there would be only one implementation. Unfortunately, stemming is an inexact science that suffers from two issues: understemming and overstemming.
Understemming is the failure to reduce words with the same ...