Chapter 22. Stopwords: Performance Versus Precision
Back in the early days of information retrieval, disk space and memory were limited to a tiny fraction of what we are accustomed to today. It was essential to make your index as small as possible. Every kilobyte saved meant a significant improvement in performance. Stemming (see Chapter 21) was important, not just for making searches broader and increasing retrieval in the same way that we use it today, but also as a tool for compressing index size.
Another way to reduce index size is simply to index fewer words. For search purposes, some words are more important than others. A significant reduction in index size can be achieved by indexing only the more important terms.
So which terms can be left out? We can divide terms roughly into two groups:
- Low-frequency terms
Words that appear in relatively few documents in the collection. Because of their rarity, they have a high value, or weight.
- High-frequency terms
Common words that appear in many documents in the index, such as
is. These words have a low weight and contribute little to the relevance score.
Of course, frequency is really a scale rather than just two points labeled low and high. We just draw a line at some arbitrary point and say that any terms below that line are low frequency and above the line are high frequency.
Which terms are low or high frequency depend on the documents themselves. The
and may be a low-frequency term if all the ...
Get Elasticsearch: The Definitive Guide now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.