Most of the examples in this chapter create a tokenizer and use it once. Because this is a common practice, the Tokenizer class lets you pass a string to tokenize into its constructor. For example, the following line creates a Tokenizer object and gives it the string to tokenize:
Tokenizer t = new Tokenizer(">give 2receive");
You can also create a tokenizer without a string and then set the string later. This approach lets you create a customized tokenizer and reuse it for many strings. For example, the CoffeeParser class in Chapter 5, “Parsing Data Languages,” creates a special tokenizer that allows spaces to appear inside words. Here is the tokenizer() method of CoffeeParser:
/** * Returns a tokenizer that ...