Feature Engineering Made Easy

The Tf-idf vectorizer

A Tf-idfVectorizer can be broken down into two components. First, the tf part, which represents term frequency, and the idf part, meaning inverse document frequency. It is a term—weighting method that has applications in information—retrieval and clustering.

A weight is given to evaluate how important a word is to a document in a corpus. Let's look into each part a little more: 

• tf: term frequency: Measures how frequently a term occurs in a document. Since documents can be different in length, it is possible that a term would appear many more times in longer documents than shorter ones. Thus, the term frequency is often divided by the document length, or the total number of terms in the document, as a way of normalization. ...