Skip to Content
Practical Predictive Analytics
book

Practical Predictive Analytics

by Ralph Winters
June 2017
Beginner to intermediate content levelBeginner to intermediate
576 pages
15h 22m
English
Packt Publishing
Content preview from Practical Predictive Analytics

Removing sparse terms

Most TDMs are initially filled with a lot of empty space. That is because every word in a corpus is indexed, and there are many words that occur so infrequently that they do not matter analytically. Removing sparse terms is a method in which we can reduce the number of terms to a manageable size, and also save space at the same time.

The removeSparseTerms() function will reduce the number of terms in the description from 268034 to 62:

dtms <- removeSparseTerms(dtm, 0.99)dim(dtms) > [1] 268034 62

As an alternative to inspect, we can also View() it in matrix form:

View(as.matrix(dtms))

Here is the output from the View command. A 1 indicates that the term occurs, and 0 indicates it did not occur:

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

Data Superstream: Analytics Engineering

Data Superstream: Analytics Engineering

Alistair Croll, Anna Filippova, Emilie Schario, Lewis Davies, Jacob Frackson, Benn Stancil, Nick Acosta, Elizabeth Caley
R: Predictive Analysis

R: Predictive Analysis

Tony Fischetti, Eric Mayor, Rui Miguel Forte
Python: Advanced Predictive Analytics

Python: Advanced Predictive Analytics

Ashish Kumar, Joseph Babcock

Publisher Resources

ISBN: 9781785886188Supplemental Content