Best practice 14 – extracting features from text data

We have worked intensively with text data in Chapter 2Exploring the 20 Newsgroups Dataset with Text Analysis Techniques, Chapter 3, Mining the 20 Newsgroups Dataset with Clustering, and Topic Modeling Algorithms, Chapter 4, Detecting Spam Email with Naive Bayes, and Chapter 5Classifying News Topics with a Support Vector Machine, where we extracted features from text based on term frequency (tf) and term frequency-inverse document frequency (tf-idf). Both methods consider each document of words (terms) a collection of words, or a bag of words (BoW), disregarding the order of words, but keeping multiplicity. A tf approach simply uses the counts of tokens, while tf-idf extends tf by assigning ...

Get Python Machine Learning By Example - Second Edition now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.