Skip to Main Content
Language Implementation Patterns
book

Language Implementation Patterns

by Terence Parr
December 2009
Intermediate to advanced content levelIntermediate to advanced
380 pages
9h 2m
English
Pragmatic Bookshelf
Content preview from Language Implementation Patterns
Pattern 2LL(1) Recursive-Descent Lexer

Purpose

Lexers derive a stream of tokens from a character stream by recognizing lexical patterns.

Lexers are also called scanners, lexical analyzers, and tokenizers. As a bonus, this pattern can recognize nested lexical structures such as nested comments (even though languages typically don’t have super-complicated lexical structure).

Discussion

The goal of the lexer is to emit a sequence of tokens. Each token has two primary attributes: a token type (symbol category) and the text associated with it. In the English language, we’ve got categories such as verbs and nouns as well as punctuation symbols such as commas and periods. All words within a particular category are said to have the same token type, though ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

Implementation Patterns

Implementation Patterns

Kent Beck

Publisher Resources

ISBN: 9781680500097Errata Page