You can customize a tokenizer in three ways: by customizing one of the tokenizer's states, by changing which state the tokenizer enters given an initial character, or by adding an entirely new state.
The preceding section shows how the CoffeeParser class creates a special tokenizer that allows spaces to appear in words. The tokenizer() method of this class retrieves a WordState object from a tokenizer t and updates it:
t.wordState().setWordChars(' ', ' ', true);
The example in Section 9.7.1 changes the state the tokenizer enters on seeing a “#” to a quote state. It uses this line:
t.setCharacterState('#', '#', t.quoteState());