R Data Analysis Projects

Why cosine similarity?

In the previous chapters we had leveraged pearson coefficient to find similarity. Why don't we use the same for text documents? Cosine similarity is non-invariant to changes in the magnitude of values. That is, if in one of the vectors we increase the value of its members, the cosine similarity will change. We need this behavior, as the vector contains the tfidf scores. A change in tfidf score means there is a change in the document. The document is no longer the same one we had before. Pearson coefficient is invariant to shifts in the vector. Hence for comparing similarities between documents, we use cosine as the metric.

Now, in our document term matrix, we have a row for each article. The columns are again the article ...