Apache Spark 2.x Cookbook

Inverse document frequency

What if we also measure how many times a term has occurred across the corpus. If a term has occurred across the corpus with the same frequency, then it is not so important to the specific document. Let's say term frequency across the corpus is DF(t, D). Average frequency per document will be as follows:

DF(t, D)/|D|

Here, |D| is the number of documents. Since the importance of a term is inversely proportional to DF(t, D), let's inverse it and take a log:

log |D|/DF(t, D)

Now it looks good, except that when the number of documents is 0, the log becomes undefined. So let's add 1 to both the numerator and the denominator to avoid the boundary condition:

(log |D| + 1 )/ (DF(t, D) + 1)

This is called inverse document ...