Sentence segmentation
Sentence segmentation is the process of splitting up a text into sentences. From its definition, it seems a straightforward process, but several difficulties can occur with it, for example, the presence of punctuation marks that can be used to indicate different things:
Streamsets Inc. has released the new Data Collector 3.5.0. One of the new features is the MongoDB lookup processor.
Looking at the preceding text, you can see that the same punctuation mark (.) is used for three different things, not just as a sentence separator. Some languages, such as Chinese for example, come with unambiguous sentence-ending markers, while others don't. So a strategy needs to be set. The quickest and dirtiest approach to locate the ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access