Unicode Explained

Naïve Processing

In old programs, character data is often processed in a naïve manner that assumes a

particular character code, typically ASCII. You might even see code like ch == 32, which

tests for a character being a space, using the ASCII code, instead of the more natural

and more portable ch == ' '.

Suppose that the variable ch contains a single character and we wish to test whether

the value is a letter. The following style (exemplified here using the C language notation)

is often used in old software:

if( ((ch >= 'A') && (ch <= 'Z')) || ((ch >= 'a') && (ch <= 'z')) ) …

Here, && means “and” and || means “or,” and the expression operates on comparisons

that test whether the character’s code number is between the code numbers of “A” and

“Z” or between the code numbers of “a” and “z.” Generally, in programming languages,

comparisons of character values operate on the code numbers of characters.

If the data contains only basic Latin letters, the naïve approach works in most cases.

The reason is that in most character codes, those letters are in alphabetic order and

consecutive—i.e., there is nothing but letters between “A” and “Z” or between “a” and

“z” in the code. However, the assumption is not correct for the EBCDIC code, as de-

scribed in Chapter 3.

A more serious problem is that the approach fails for letters with diacritic marks, or for

other than basic Latin letters in general. It would ...