Natural Language Processing: Python and NLTK

Evaluate a language model through perplexity

The nltk.model.ngram module in NLTK has a submodule, perplexity(text). This submodule evaluates the perplexity of a given text. Perplexity is defined as 2**Cross Entropy for the text. Perplexity defines how a probability model or probability distribution can be useful to predict a text.

The code for evaluating the perplexity of text as present in the nltk.model.ngram module is as follows:

def perplexity(self, text):
"""
        Calculates the perplexity of the given text.
        This is simply 2 ** cross-entropy for the text.

        :param text: words to calculate perplexity of
        :type text: list(str)
"""

        return pow(2.0, self.entropy(text))