
most common situation for its presence is that data has been converted from UTF-16
or UTF-32 without removing BOM. (In UTF-8, BOM is the octet sequence EF BB BF.)
The BOM is to be treated as indicating the byte order only, not as part of the data.
Previously, code point U+FEFF was defined to have the meaning of a zero width no-
break space (ZWNBSP), too, and it could appear in the middle of text, too. This usually
did not cause problems, but such usage has now been deprecated. In theory, when you
detect U+FEFF at the start of UTF-8 data, you cannot know for sure whether it is meant
to be a byte order mark or just a no-break space as part of the data proper. In practice,
this seldom makes a difference, since an initial no-break space doesn’t really matter.
However, if you concatenate files, for example, it might matter.
If U+FEFF is encountered within text, it should be treated as ZWNBSP, which acts as
invisible “glue” that prevents a line break between characters. However, you should
not use it that way in new data; the recommended “glue” character is word joiner
U+2060. Unicode implementations are allowed to convert U+FEFF (inside data) to
U+2060.
There is no way in Unicode to change the byte order within a file. If U+FEFF appears
anywhere else except at the start of character data, it must be interpreted according to
the no-break space semantics (or not be interpreted at all).
Due to the stability ...