Deep Learning Essentials

Learning rate

Learning rate is critical for fast convergence. Learning rate determines how much at each iteration-model parameter should be updated proportionally to the Gradient of the loss function:

Choosing a proper learning rate can be difficult. A learning rate that is too small leads to very slow training and convergence, while a learning rate that is too large can lead to overshoot and bounce, and cause the loss function to fluctuate around the minimum or even to diverge.

There are a few ways of setting up learning rate:

• Constant: Learning rate remains the same for all epochs
• Step decay: Learning rate decays every t number of epochs ...