Unicode Explained

most common situation for its presence is that data has been converted from UTF-16

or UTF-32 without removing BOM. (In UTF-8, BOM is the octet sequence EF BB BF.)

The BOM is to be treated as indicating the byte order only, not as part of the data.

Previously, code point U+FEFF was defined to have the meaning of a zero width no-

break space (ZWNBSP), too, and it could appear in the middle of text, too. This usually

did not cause problems, but such usage has now been deprecated. In theory, when you

detect U+FEFF at the start of UTF-8 data, you cannot know for sure whether it is meant

to be a byte order mark or just a no-break space as part of the data proper. In practice,

this seldom makes a difference, since an initial no-break space doesn’t really matter.

However, if you concatenate files, for example, it might matter.

If U+FEFF is encountered within text, it should be treated as ZWNBSP, which acts as

invisible “glue” that prevents a line break between characters. However, you should

not use it that way in new data; the recommended “glue” character is word joiner

U+2060. Unicode implementations are allowed to convert U+FEFF (inside data) to

U+2060.

There is no way in Unicode to change the byte order within a file. If U+FEFF appears

anywhere else except at the start of character data, it must be interpreted according to

the no-break space semantics (or not be interpreted at all).

Due to the stability ...