Hands-On Deep Learning with Apache Spark

Tokenizers

Tokenization means defining what a word is in NLP ML algorithms. Given a text, tokenization is the task of cutting it down into pieces, called tokens, while at the same time also removing particular characters (such as punctuation or delimiters). For example, given this input sentence in the English language:

To be, or not to be, that is the question

The result of tokenization would produce the following 11 tokens:

To be or or not to be that is the question

One big challenge with tokenization is about what the correct tokens to use are. In the previous example, it was easy to decide: we cut down on white spaces and removed all the punctuation characters. But what if the input text isn't in English? For some other languages, such ...