Machine Learning with Spark - Second Edition

Extracting the tf-idf features from the 20 Newsgroups dataset

To illustrate the concepts in this chapter, we will use a well-known text dataset called 20 Newsgroups; this dataset is commonly used for text-classification tasks. This is a collection of newsgroup messages posted across 20 different topics. There are various forms of data available. For our purposes, we will use the bydate version of the dataset, which is available at http://qwone.com/~jason/20Newsgroups.

This dataset splits up the available data into training and test sets that comprise 60 percent and 40 percent of the original data, respectively. Here, the messages in the test set occur after those in the training set. This dataset also excludes some of the message headers ...