O'Reilly logo

Building Parsers with Java™ by Steven John Metsker

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

9.2. Acquiring a Tokenizer

You have several choices regarding how you acquire a tokenizer to use for your parsers. For one thing, you can use a tokenizer that comes with the freely available Java software development kit (JDK). However, as of this writing, these tokenizers lack features that the languages in this book require. To overcome these limitations, a new tokenizer was developed for this book, and this chapter explains how to use it.

This discussion opens other approaches to acquiring a tokenizer. You can use the tokenizer that comes with this book as is, without changing its code. The Tokenizer class in sjm.parse.tokens is highly customizable, and you may find that it meets all your needs for tokenization. Another choice is to compare ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required