Intelligent Projects Using Python

Perplexity for language models

The perplexity metrics are used to evaluate the usefulness of a language model. Let's assume that we have trained a language model on a training corpus and let the learned probability model over sentences or text be P(.). The perplexity of the model P(.) is evaluated on a test set corpus drawn from the same population as that of the training corpus. If we represent the test set corpus by the M words, say (w₁, w₂, w₃, . . . . . , w_M), then the perplexity of the model over the test set sequence is represented by the following:

The expression for H as shown measures the per-word uncertainty:

As per the language ...