December 2008
Intermediate to advanced
912 pages
33h 22m
English
Content preview from CJKV Information Processing, 2nd Edition
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
608
|
Chapter 9: Information Processing Techniques
Natural Language Processing
Attempting to derive meaning from text requires natural language processing. e most
fundamental level of natural language processing is the ability to parse the text into words.
e words themselves can be further broken down through morphological analysis. Ap-
plications for this technology include spelling and grammar checkers, along with tagging
and indexing. e following sections will explore natural language-processing techniques,
along with some of its applications.
Word Parsing and Morphological Analysis
Parsing most Western-language sentences into their constituent words is a somewhat
trivial operation due to the intervening—and necessary—spaces between words. Other
than the occasional punctuation and dealing with case issues, there is very little diculty.
Most CJKV sentences, however, oer signicant challenges in this area of information
processing. e ultimate goal of parsing sentences into their component words is for com-
mon purposes such as determining the key words of a document, which is called tagging.
Tagging is useful for categorizing or indexing documents based on their content, and it is
a key function for search engines.
Most CJKV sentences, such as those for Chinese and Japanese, include no intervening
spaces between words. Korean, on the other hand, does use spaces ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.