Hands-On Machine Learning for Algorithmic Trading

Document-term matrix with sklearn

The scikit-learn preprocessing module offers two tools to create a document-term matrix. CountVectorizer uses binary or absolute counts to measure the term frequency tf(d, t) for each document d and token t.

TfidFVectorizer, in contrast, weighs the (absolute) term frequency by the inverse document frequency (idf). As a result, a term that appears in more documents will receive a lower weight than a token with the same frequency for a given document but lower frequency across all documents. More specifically, using the default settings, tf-idf(d, t) entries for the document-term matrix are computed as tf-idf(d, t) = tf(d, t) x idf(t):

Here n_d is the number of documents and df(d, t) the document frequency ...