Machine Learning Algorithms - Second Edition

Word tokenizing

The simplest way to tokenize a sentence into words is provided by the TreebankWordTokenizer class; however, this has some limitations:

from nltk.tokenize import TreebankWordTokenizersimple_text = 'This is a simple text.'tbwt = TreebankWordTokenizer()print(tbwt.tokenize(simple_text))['This', 'is', 'a', 'simple', 'text', '.']complex_text = 'This isn\'t a simple text'print(tbwt.tokenize(complex_text))['This', 'is', "n't", 'a', 'simple', 'text']

As you can see, in the first case, the sentence has been correctly split into words, keeping the punctuation separate (this is not a real issue because it can be removed in a second step). However, in the complex example, the contraction isn't has been split into is and n't. Unfortunately, ...