Hands-On Generative Adversarial Networks with Keras

Learning rate schedule

Choose learning rates according to the problem at hand.

One of the challenges in training neural networks is setting the proper learning rate. Within the GAN framework, wherein one looks for an equilibrium between the discriminator and the generator networks, this challenge gets exacerbated given the presence of a learning rate for each network and the dependencies between losses.

For discriminator and generator networks of equal size, one expects the discriminator to have some advantage over the generator such that the updates provided by the discriminator to the generator are useful. This can be achieved by setting the discriminator’s learning rate to be slightly higher than the generator, or by performing more discriminator ...