Machine Learning Algorithms - Second Edition

Divergence measures between two probability distributions

Let's suppose you have a discrete data generating process p_data(x) and a model that outputs a probability mass function q(x). In many machine learning tasks, the goal is to tune up the parameter so that q(x) becomes as similar to p_data as possible. A very useful measure is the Kullback-Leibler divergence:

This quantity (also known as information gain) expresses the gain obtained by using the approximation q(x) instead of the original data generating process. It's immediate to see that if q(x) = p_data(x) ⇒ D_KL(p_data||q) = 0, while it's greater than 0 (unbounded) when there's a mismatch. ...