Understanding SBD rules of LingPipe's HeuristicSentenceModel class
There are other rules that can be used to perform SBD. LingPipe's HeuristicSentenceModel class uses a series of token rules to perform SBD. We will present them here, as they provide insight into what rules can be useful.
This class uses three sets of tokens and two flags to assist in the process:
- Possible Stops: This is a set of tokens that can be the last token of a sentence
- Impossible Penultimates: These tokens cannot be the second to last token in a sentence
- Impossible Starts: This is a set of tokens that cannot be used to start a sentence
- Balance Parentheses: This flag indicates that a sentence should not be terminated until all matching parentheses are matched in that sentence ...
Get Natural Language Processing with Java now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
