Building Parsers with Java™

9.11. Token Strings

The package sjm.parse.tokens uses a TokenString class to hold the results of tokenizing a string. A TokenString is similar to a String, but it contains a series of tokens rather than a series of characters. Like String objects, TokenString objects are immutable, meaning that they cannot change after they are created. Figure 9.17 shows the TokenString class.

Figure 9.17. The TokenString class. A TokenString is essentially an array of Tokens. Like String, TokenString is immutable, so there is never a need to copy a TokenString.

The TokenAssembly class hides the fact that it relies on class TokenString. The TokenStringSource class, ...