The Definitive ANTLR 4 Reference, 2nd Edition

Chapter 12Wielding Lexical Black Magic

In this part of the book, we’ve learned some advanced skills. We know how to execute arbitrary code while parsing, and we can alter syntax recognition with semantic predicates. Now it’s time to put those skills to work solving some challenging language recognition problems but in the lexer this time, not the parser.

In my experience, if a language problem is hard to solve, most of the head-scratching occurs in the lexer (well, with the exception of C++, which is hard all over). That’s counterintuitive because the lexer rules we’ve seen so far have been pretty simple, such as identifiers, integers, and arithmetic expression operators. But, consider the harmless-looking two-character Java sequence: >> ...