The Definitive ANTLR 4 Reference, 2nd Edition

Tokens and Token Factories

Lexers break up character streams into Token objects, and parsers apply grammatical structure to the stream of tokens. Generally, we think of tokens as immutable after construction in the lexer, but sometimes we need to alter token fields after we’ve created them. For example, the token streams like to set the token index of the tokens as they go by. To support this, ANTLR uses interface WritableToken, which is a kind of Token with “setter” methods. Finally, we have CommonToken that implements a full-featured token.

We usually don’t need to define our own kind of tokens, but it’s a useful capability. Here’s a sample ...