Detecting tokens (words) using Java

One of the most common tasks that a data scientist needs to do using text data is to detect tokens from it. This task is called tokenization . Although "token" can mean a word, symbol, phrase or any other meaningful text unit, in this chapter, we will consider words as tokens since a word is a reasonable text unit to deal with. However, the notion of word tokens varies from person to person; some need words only, some prefer symbols to be omitted during detection, while some want to keep punctuation in the words for getting more context. Based on the variety of needs, in this recipe, we will be using three different techniques that, when applied to the same string, yield three different results. The techniques ...

Get Java Data Science Cookbook now with the O’Reilly learning platform.

O’Reilly members experience live online training, plus books, videos, and digital content from nearly 200 publishers.