Programming Languages: Concepts and Implementation

3.2 Scanning

For purposes of scanning, the valid lexical units of a program are called lexemes (e.g., ). The first step of scanning (also referred to as lexical analysis) is to parcel the characters (from the alphabet Σ) of the string representing the line of code into lexemes. Lexemes can be formally described by regular expressions and regular grammars. Lexical analysis is the process of determining if a string (typically of a programming language) is lexically valid— that is, if all of the lexical units of the string are lexemes.

Programming languages must specify how the lexical units of a program are delimited. There are a variety of methods ...