Deep Learning Essentials

Weighting the terms tf-idf

In most languages, some words tend to appear more often than others but may not contain much differentiative information regarding judging the similarity of two documents. Examples are words such as is, the, and a, which are all very common in English. If we only consider their raw frequency as we did in the previous session, we might not be able to effectively differentiate between different classes of documents or retrieve the similar documents that match the core content.

One approach to tackle this problem is called term frequency and inverse document frequency (tf-idf). Like its name, it takes into account two terms: term frequency (tf) and inverse document frequency (idf).

With tf, tf(t,d), the simplest choice ...