Mastering Machine Learning Algorithms

RMSProp

RMSProp was proposed by Hinton as an adaptive algorithm, partially based on the concept of momentum. Instead of considering the whole gradient vector, it tries to optimize each parameter separately to increase the corrections of slowly changing weights (that probably need more drastic modifications) and decreasing the update magnitudes of quickly changing ones (which are normally the more unstable). The algorithm computes the exponentially weighted moving average of the changing speed of every parameter considering the square of the gradient (which is insensitive to the sign):

The weight update is then performed, as follows:

The parameter ...