Skip to Main Content
Unicode Explained
book

Unicode Explained

by Jukka K. Korpela
June 2006
Beginner content levelBeginner
688 pages
26h 18m
English
O'Reilly Media, Inc.
Content preview from Unicode Explained
most common situation for its presence is that data has been converted from UTF-16
or UTF-32 without removing BOM. (In UTF-8, BOM is the octet sequence EF BB BF.)
The BOM is to be treated as indicating the byte order only, not as part of the data.
Previously, code point U+FEFF was defined to have the meaning of a zero width no-
break space (ZWNBSP), too, and it could appear in the middle of text, too. This usually
did not cause problems, but such usage has now been deprecated. In theory, when you
detect U+FEFF at the start of UTF-8 data, you cannot know for sure whether it is meant
to be a byte order mark or just a no-break space as part of the data proper. In practice,
this seldom makes a difference, since an initial no-break space doesn’t really matter.
However, if you concatenate files, for example, it might matter.
If U+FEFF is encountered within text, it should be treated as ZWNBSP, which acts as
invisible “glue” that prevents a line break between characters. However, you should
not use it that way in new data; the recommended “glue” character is word joiner
U+2060. Unicode implementations are allowed to convert U+FEFF (inside data) to
U+2060.
There is no way in Unicode to change the byte order within a file. If U+FEFF appears
anywhere else except at the start of character data, it must be interpreted according to
the no-break space semantics (or not be interpreted at all).
Due to the stability ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

Unicode Demystified

Unicode Demystified

Richard Gillam
Fonts & Encodings

Fonts & Encodings

Yannis Haralambous
Core Java™ Data Objects

Core Java™ Data Objects

Sameer Tyagi, Keiron McCammon, Michael Vorburger, Heiko Bobzin

Publisher Resources

ISBN: 059610121XCatalog PageErrata