Best practice 14 – extracting features from text data

We have worked intensively with text data in Chapter 2, Exploring the 20 Newsgroups Dataset with Text Analysis Techniques, Chapter 3, Mining the 20 Newsgroups Dataset with Clustering, and Topic Modeling Algorithms, Chapter 4, Detecting Spam Email with Naive Bayes, and Chapter 5, Classifying News Topics with a Support Vector Machine, where we extracted features from text based on term frequency (tf) and term frequency-inverse document frequency (tf-idf). Both methods consider each document of words (terms) a collection of words, or a bag of words (BoW), disregarding the order of words, but keeping multiplicity. A tf approach simply uses the counts of tokens, while tf-idf extends tf by assigning ...

Get Python Machine Learning By Example - Second Edition now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.