Programming Perl, 4th Edition

Graphemes and Normalization

We already mentioned characters like latin small letter dz that occupy one codepoint but that may look like two characters to the end user. The opposite situation also exists and is much more common. That is, a single user-visible character (a grapheme) can require more than a single codepoint to represent it. Think of a letter plus one or more diacritics, like both és in “résumé”. Those might each be one codepoint, or two. It’s even possible that one é of them is a letter codepoint but the other is a letter followed by a combining mark. By design, you can never tell the difference just by looking at them because they are considered canonically equivalent. This has serious ramifications for almost all text handling, and it very nearly contradicts what we said earlier about glyphs not being important. In this particular sense they are the most important.

Combining characters are used to change an “n” into an “ñ”, a “c” into a “ç”, an “o” into an “ô”, or a “u” into a “ǘ”. The first three transformations require one combining mark, while the last one requires two combining marks. In fact, there’s no limit to these. You can keep piling them on as long as you’d like, and you can create things people have never seen before.

All this requires quite a bit of serious rethinking and rewriting of all kinds of software. Just think about what the font system has to do. (No, giving up is not a valid option.) Your own programs that process text may need serious overhauling. ...