The PTBTokenizer class uses rules to perform SBD and has a variety of tokenization options. The constructor for this class possesses three parameters:
- A Reader class that encapsulates the text to be processed
- An object that implements the LexedTokenFactory interface
- A string holding the tokenization options
These options allow us to specify the text, the tokenizer to be used, and any options that we may need to use for a specific text stream.
In the following code sequence, an instance of the StringReader class is created to encapsulate the text. The CoreLabelTokenFactory class is used with the options left as null for this example:
PTBTokenizer ptb = new PTBTokenizer(new StringReader(paragraph), new CoreLabelTokenFactory(), ...
