token — A unit of information
phrase db._phrase |
| | | | )*
token identifies a unit of information.
Usually, “tokens” are the result of some processing pass
that has performed lexical analysis and divided a data set into the
smallest units of information used for subsequent processing.
Exactly what constitutes a token varies by context.
<article xmlns='http://docbook.org/ns/docbook'> <title>Example token</title> <para>In parsing, line ends are turned into the <token>CRLF</token>, all other whitespace becomes <token>WHITESP</token>. </para> </article>