9.7. Tokenizer States
A tokenizer state is an object that can return a token from an input reader. The Tokenizer class in sjm.parse.tokens uses, by default, the classes that Figure 9.6 shows.
Figure 9.6. The TokenizerState hierarchy. This diagram shows the hierarchy of TokenizerState in sjm.parse.tokens. A Tokenizer object can use these states to consume tokens from a string.
Each subclass implements the nextToken() method of the abstract class TokenizerState, which Figure 9.7 shows.
Figure 9.7. The TokenizerState class. TokenizerState is an abstract class that defines the minimal behavior of an object that can read characters and build a corresponding ...