Building Parsers with Java™

9.10. The TokenStringSource Utility

The TokenStringSource class in sjm.parse.tokens is a utility that helps divide an input stream into lines that a parser can parse one at a time. Figure 9.16 shows this class.

Figure 9.16. The TokenStringSource class. The TokenStringSource nextTokenString() method returns all the tokens up to the specified delimiter as a TokenString.

This class provides an enumeration over the tokens of a tokenizer. Each call to nextTokenString() results in a TokenString that contains all the tokens up to a specified delimiter. This is useful for languages that consist of statements that are separated by a delimiter, such as “ ...