Skip to Main Content
Natural Language Processing with Java - Second Edition
book

Natural Language Processing with Java - Second Edition

by Richard M. Reese, AshishSingh Bhatia
July 2018
Beginner to intermediate content levelBeginner to intermediate
318 pages
7h 49m
English
Packt Publishing
Content preview from Natural Language Processing with Java - Second Edition

Using the PTBTokenizer class

The PTBTokenizer class uses rules to perform SBD and has a variety of tokenization options. The constructor for this class possesses three parameters:

  • A Reader class that encapsulates the text to be processed
  • An object that implements the LexedTokenFactory interface
  • A string holding the tokenization options

These options allow us to specify the text, the tokenizer to be used, and any options that we may need to use for a specific text stream.

In the following code sequence, an instance of the StringReader class is created to encapsulate the text. The CoreLabelTokenFactory class is used with the options left as null for this example:

PTBTokenizer ptb = new PTBTokenizer(new StringReader(paragraph),  new CoreLabelTokenFactory(), ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

Natural Language Processing with Java Cookbook

Natural Language Processing with Java Cookbook

Richard M. Reese, Richard M Reese
Natural Language Processing in Action

Natural Language Processing in Action

Cole Howard, Hobson Lane, Hannes Hapke
Natural Language Processing with Python

Natural Language Processing with Python

Steven Bird, Ewan Klein, Edward Loper

Publisher Resources

ISBN: 9781788993494Supplemental Content