A Rational Statistical Parser
Abstract: A model of syntactic parsing that combines elements of information and probability theory is proposed. The model assigns probability and entropy scores to parse trees: trees with higher probabilities are preferred while trees with higher entropies are penalized. This model is argued to be psycholinguistically motivated by means of rational analysis. Using a grammar extracted from the Penn Treebank, the implemented model was evaluated on the section 23 of the corpus. The results present a modest but general improvement in almost all types of phenomena analyzed, suggesting that the inclusion of entropy is beneficial during parsing and that, given our formulation, its relevance ...
Get Natural Language Processing and Cognitive Science now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
