Python Machine Learning By Example - Second Edition

Best practice 14 – extracting features from text data

We have worked intensively with text data in Chapter 2, Exploring the 20 Newsgroups Dataset with Text Analysis Techniques, Chapter 3, Mining the 20 Newsgroups Dataset with Clustering, and Topic Modeling Algorithms, Chapter 4, Detecting Spam Email with Naive Bayes, and Chapter 5, Classifying News Topics with a Support Vector Machine, where we extracted features from text based on term frequency (tf) and term frequency-inverse document frequency (tf-idf). Both methods consider each document of words (terms) a collection of words, or a bag of words (BoW), disregarding the order of words, but keeping multiplicity. A tf approach simply uses the counts of tokens, while tf-idf extends tf by assigning ...