Java Data Science Cookbook

Detecting tokens (words) using Java

One of the most common tasks that a data scientist needs to do using text data is to detect tokens from it. This task is called tokenization . Although "token" can mean a word, symbol, phrase or any other meaningful text unit, in this chapter, we will consider words as tokens since a word is a reasonable text unit to deal with. However, the notion of word tokens varies from person to person; some need words only, some prefer symbols to be omitted during detection, while some want to keep punctuation in the words for getting more context. Based on the variety of needs, in this recipe, we will be using three different techniques that, when applied to the same string, yield three different results. The techniques ...