Python Deep Learning - Second Edition

N-grams

The inference of the probability of a long sequence, say w₁, ..., w_m, is typically infeasible. Calculating the joint probability of P(w₁, ... , w_m) would be done by applying the following chain rule:

The probability of the later words given the earlier words would be especially difficult to estimate from the data. That's why this joint probability is typically approximated by an independence assumption that the ith word is only dependent on the n-1 previous words. We'll only model the joint probabilities of combinations of n sequential words, called n-grams. For example, in the phrase the quick brown fox, we have the following n-grams: ...