O'Reilly logo

Building Parsers with Java™ by Steven John Metsker

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

9.3. Tokenizers in Standard Java

The standard Java libraries include two tokenizers: StringTokenizer in java.util and StreamTokenizer in java.io.

The StringTokenizer class does not parse numbers, and it allows little customization. This tokenizer is suitable only for simple tokenization, and this book does not discuss it further.

The StreamTokenizer class is more customizable than StringTokenizer but lacks some desirable features. In particular, StreamTokenizer in Java 1.1.7 does not provide

  • A Token class to encapsulate token results

  • Customization of how to recognize numbers

  • The ability to define new token types

  • Differentiation of allowable characters for the start of a word from allowable characters within a word

  • Handling of multicharacter symbols ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required