Unicode Explained

Auto-Detecting the Encoding

The encoding of data should be explicitly told to any potential recipient. In particular,

on the Internet, special headers have been designed for informing the encoding of a

web page or a message, as described in Chapter 10.

However, quite often we are faced with data that is known or suspected to be in a

Unicode encoding, but we don’t know which. Moreover, we might not wish to trust

the indication of the encoding without performing some simple checks. Table 6-4

presents basic methods for guessing the encoding from the first few octets of data.

Beware that the result is at best a good guess. The second column shows how the first

few octets, shown in column one in hexadecimal, look when interpreted according to

the ISO-8859-1 encoding (which is what many simple editors and software for dumping

data in text format use by default). If the data starts in some other way, it could still be

in a Unicode encoding, but without a byte order mark.

Table 6-4. Heuristics for detecting Unicode encoding

First octets of data ISO-8859-1 view Probable encoding

FE FF þÿ UTF-16

FF FE ÿþ UTF-16LE

00 00 FE FF (nul)(nul)þÿ UTF-32

00 00 FF FE (nul)(nul)ÿþ UTF-32LE

EF BB BF  UTF-8

0E FE FF (Ctrl-N)þÿ SCSU

DD 73 73 73 Ýsss UTF-EBCDIC

Choosing an Encoding

The

Unicode standard explicitly says that the Unicode Consortium “fully endorses the

use of any of the three Unicode encoding forms ...