O'Reilly logo

Building Parsers with Java™ by Steven John Metsker

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 9. Advanced Tokenizing

Tokenizing a string means dividing the string into logical chunks, or tokens. This lets you define a grammar in terms of token terminals, something that is much simpler than defining a grammar in terms of every character in an input stream. Simpler grammars translate into simpler parsers, so tokenizing reduces the complexity of a parser. This chapter explains how tokenizing works and describes techniques for controlling in detail how sequences of characters form tokens.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required