Hands-On Unsupervised Learning with Python

Wasserstein GANs

Given a probability distribution p(x), the set D_p = {x : p(x) > 0} is called support. If two distributions, p(x) and q(x), have disjointed supports (that is, D_p ∩ D_q = {∅}), the Jensen-Shannon divergence becomes equal to log(2). This means that the gradient is null, and no corrections can happen anymore. In a generic scenario where a GAN is involved, it's extremely unlikely that p_g(x) and p_data are fully overlapped (however, you can expect a minimum overlap); therefore, the gradients are very small, and so are the updates to the weights. Such a problem could block the training process and trap the GAN in a suboptimal configuration that cannot be escaped. For this reason, Arjovsky, Chintala, and Bottou (in Wasserstein GAN, ...