Hands-On Deep Learning Algorithms with Python

Overcoming the limitations of Adagrad using RMSProp

Similar to Adadelta, RMSProp was introduced to combat the decaying learning rate problem of Adagrad. So, in RMSProp, we compute the exponentially decaying running average of gradients as follows:

Instead of taking the sum of the square of all the past gradients, we use this running average of gradients. This means that our update equation becomes the following:

It is recommended to assign a value of learning to 0.9. Now, we will learn how to implement RMSProp in Python.

First, we need to ...