Java: Data Science Made Easy

Indexing and Apache Lucene

Previously, we looked at how to implement a simple search engine, but it will not scale well with the number of documents.

First of all, it requires a comparison of the query with each and every document in our collection, and it becomes very time-consuming as it grows. Most of the documents, however, are not relevant to the query, and only a small fraction are. We can safely assume that, if a document is relevant to a query, it should contain at least one word from it. This is the idea behind the Inverted Index data structure: for each word it keeps track of the documents that contain it. When a query is given, it can quickly find the documents that have at least one term from it.

There also is a memory problem: ...