O'Reilly logo

Building Parsers with Java™ by Steven John Metsker

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

11.3. New Token Types

A tokenizer is responsible for breaking text into logical elements. Your idea of what a “logical” element is may differ from the defaults provided by class Tokenizer in sjm.parse.tokens. For example, you may want to distinguish real numbers from imaginary numbers, or HTML tags from other kinds of text, or reserved and nonreserved words. Often, when you create a new type of token, you will create one or more new Terminal types to recognize it.

One important reason for changing a tokenizer is to avoid ambiguity. Section 4.2.2, “Terminal Ambiguity,” shows the ambiguity of the following grammar:

query  = (Word | volume)* '?'; 
volume = "cups" | "gallon" | "liter";

This grammar is ambiguous because a word such as "cups" is both ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required