Python Deep Learning Projects

Creating the vectorizer

Now, let's initialize the TF-IDF vectorizer and define a few parameters:

• min_df: When building the vocabulary, ignore terms that have a document frequency strictly lower than the given threshold
• ngram_range: Configures our vectorizer to capture n-words at a time
• norm: This is used to normalize term vectors using L1 or L2 norms
• encoding: Handles Unicode characters

There are many more parameters that you can look into, configure, and play around with:

from sklearn.feature_extraction.text import TfidfVectorizervectorizer = TfidfVectorizer(min_df=0, ngram_range=(2, 4), strip_accents='unicode',norm='l2' , encoding='ISO-8859-1')

Now, we train the model on the questions:

# We create an array for our train data set (questions) ...