O'Reilly logo

Building Parsers with Java™ by Steven John Metsker

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

9.7. Tokenizer States

A tokenizer state is an object that can return a token from an input reader. The Tokenizer class in sjm.parse.tokens uses, by default, the classes that Figure 9.6 shows.

Figure 9.6. The TokenizerState hierarchy. This diagram shows the hierarchy of TokenizerState in sjm.parse.tokens. A Tokenizer object can use these states to consume tokens from a string.

Each subclass implements the nextToken() method of the abstract class TokenizerState, which Figure 9.7 shows.

Figure 9.7. The TokenizerState class. TokenizerState is an abstract class that defines the minimal behavior of an object that can read characters and build a corresponding ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required