Python Data Analysis Cookbook

Implementing a basic terms database

As you know, natural language processing has many applications:

• Full text search as implemented by commercial and open source search engines
• Clustering of documents
• Classification, for example to determine the type of text or the sentiment in the context of a product review

To perform these tasks, we need to calculate features such as TF-IDF scores (refer to Stemming, lemmatizing, filtering, and TF-IDF scores). Especially, with large datasets, it makes sense to store the features for easy processing. Search engines use inverted indices, which map words to web pages. This is similar to the association table pattern (refer to Implementing association tables).

We will implement the association table pattern with three ...