Unicode Explained

browser in the processing of some HTML and CSS constructs, in a manner that violates

their specifications. See http://www.quirksmode.org/ for more explanations.

The bottom line is that if you wish to serve an XHTML document on the Web, it is

best to make it UTF-8 encoded (so that you can omit the XML declaration). If that is

not possible, you should use actual HTTP headers to specify the encoding.

Heuristics of detecting encoding

When none of the methods just described has been used to specify the character en-

coding, the browser has to make a guess or give up. Browsers generally try to apply

heuristic reasoning rules to deduce the encoding. At http://www.i18nfaq.com/char

det.html, you can find a Java version of the heuristic code used in Mozilla.

Remarkably often, browsers make a right guess. It is in principle impossible to deter-

mine the encoding of text from the text alone, but in practice, you can often guess right

even using automated tools. Different encodings have special properties and known

areas of application. More important, a browser knows what to expect.

HTML documents can be expected to start (aside from a possible BOM) with a coded

representation of characters from the ASCII repertoire, even if they then go on to present

a document body containing a wide range of Unicode. Moreover, there are specific

constructs (like a document type declaration and HTML tags) to be expected. ...