CJKV Information Processing, 2nd Edition

610

|

Chapter 9: Information Processing Techniques

hangul instead of hanja, and the intervening spaces help in the eort to parse them into

the constituent parts.

Fujitsu Laboratories in Japan had developed a Japanese morphological analyzer called

Breakfast that had the ability to parse Japanese text into morphemes, and had a customi-

zable POS

*

(part-of-speech) system. is customizable feature enabled Breakfast to use

the dictionaries of JUMAN ( juman)

†

and ChaSen ( chasen),

‡

both of which

still seem to be available in some form, though Breakfast seems to have gone away. An-

other called Sumomo ( sumomo), developed by NTT, also seems to have gone away.

Chances are, Breakfast and Sumomo were either sold and renamed, or simply renamed.

In addition to JUMAN and ChaSen, other more current Japanese morphological analyz-

ers include MeCab ( mekabu)

§

and KAKASI (Kanji Kana Simple Inverter).

¶

Without a doubt, Basis Technology’s Rosette Base Linguistics for Asian Languages, which

provides a morphological analyzer that supports Chinese, Japanese, and Korean, is one

of the top-performing libraries of its kind.

**

e fact that Google and Amazon use it states

something about its eectiveness. Its abilities include segmentation, tokenization, noun

decompounding, part-of-speech tagging, sentence boundary detection, and other analyz-

ing functions. ey also provide related soware, ...