Hands-On Machine Learning for Algorithmic Trading

How to implement LSI using sklearn

We will illustrate the application of LSI using the BBC article data that we introduced in the last chapter because it is small enough to permit quick training and allow us to compare topic assignments to category labels. See the latent_semantic_indexing notebook for additional implementation details:

1. We begin by loading the documents and creating a train and (stratified) test set with 50 articles.
2. Then, we vectorize the data using TfidfVectorizer to obtain weighted DTM counts and filter out words that appear in less than 1% or more than 25% of the documents, as well as generic stopwords, to obtain a vocabulary of around 2,900 words:

vectorizer = TfidfVectorizer(max_df=.25, min_df=.01,stop_words='english', ...