5.4. A Tokenizing Problem
The coffee grammar accepts coffee names, roasts, and countries as Word terminals. This creates a problem if any of these “words” contains a blank. For example, a coffee name might be “Toasty Rita,” from Costa Rica. By default, the class Tokenizer in sjm.parse.tokens treats a blank as the end of a word. When tokenizing the text
Toasty Rita, Italian, Costa Rica, 9.95
a default tokenizer would return Toasty as a Word, followed by Rita as a Word. After the first word, the grammar will be looking for a comma and not another word, and a parser generated from the grammar will fail to match the input text.
One solution is to ask the tokenizer to allow blanks to occur inside words. The following code snippet creates such a ...
Get Building Parsers with Java™ now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
