July 2018
Beginner to intermediate
318 pages
7h 49m
English
The PTBTokenizer class uses rules to perform SBD and has a variety of tokenization options. The constructor for this class possesses three parameters:
These options allow us to specify the text, the tokenizer to be used, and any options that we may need to use for a specific text stream.
In the following code sequence, an instance of the StringReader class is created to encapsulate the text. The CoreLabelTokenFactory class is used with the options left as null for this example:
PTBTokenizer ptb = new PTBTokenizer(new StringReader(paragraph), new CoreLabelTokenFactory(), ...