Machine Learning in Java - Second Edition

Pre-processing text data

Once we initialize an iterator that will go through the data, we need to pass the data through a sequence of transformations as described at the beginning of this section. Mallet supports this process through a pipeline and a wide variety of steps that could be included in a pipeline, which are collected in the cc.mallet.pipe package. Some examples are as follows:

• Input2CharSequence: This is a pipe that can read from various kinds of text sources (either URL, file, or reader) into CharSequence
• CharSequenceRemoveHTML: This pipe removes HTML from CharSequence
• MakeAmpersandXMLFriendly: This converts & into &amp in tokens of a token sequence
• TokenSequenceLowercase: This converts the text in each token in the token sequence ...