Building Parsers with Java™

9.8. Setting a Tokenizer's Source

Most of the examples in this chapter create a tokenizer and use it once. Because this is a common practice, the Tokenizer class lets you pass a string to tokenize into its constructor. For example, the following line creates a Tokenizer object and gives it the string to tokenize:

Tokenizer t = new Tokenizer(">give 2receive"); 

You can also create a tokenizer without a string and then set the string later. This approach lets you create a customized tokenizer and reuse it for many strings. For example, the CoffeeParser class in Chapter 5, “Parsing Data Languages,” creates a special tokenizer that allows spaces to appear inside words. Here is the tokenizer() method of CoffeeParser:

 /** * Returns a tokenizer that ...