Unicode Explained

Conversions between UTF-8 and UTF-16 are best performed via an intermediate rep-

resentation that corresponds to UTF-32. This does not require the creation of an actual

UTF-32 coded representation of the file or data stream. Instead, you can operate on

just the code points: read code units from UTF-8 (or UTF-16) encoded data as much

as needed to determine the Unicode code number that they represent, and then encode

this number in UTF-16 (or, respectively, in UTF-8).

Other Encodings

In addition to the encodings defined in the Unicode and ISO 10646 standards, there

are several encodings that have been used or at least proposed for Unicode data. We

will discuss some of them, summarized in Table 6-3 in alphabetic order by name. For

completeness, the table contains also the previously discussed UTF and UCS encod-

ings.

Table 6-3. Encodings used for Unicode data

Name of encoding Nature and usage of the encoding

Base64 General purpose encoding, used as “transfer encoding”

BOCU-1 A compression scheme for Unicode; not used much

CESU-8 A mixture of UTF-8 and UTF-16 for special usage

GB18030 “Chinese Unicode,” technically a separate character code

Modified UTF-8 Used in Java programming; CESU-8 with an additional change

Punycode An encoding for Internationalized Domain Names (IDN)

Quoted Printable Transfer encoding especially for email

SCSU A standardized compression scheme for Unicode; little used

UCS-2