Mastering Machine Learning Algorithms

AdaDelta

AdaDelta is an algorithm (proposed in ADADELTA: An Adaptive Learning Rate Method, Zeiler M. D., arXiv:1212.5701 [cs.LG]) in order to address the main issue of AdaGrad, which arises to managing the whole squared gradient history. First of all, instead of the accumulator, AdaDelta employs an exponentially weighted moving average, like RMSProp:

However, the main difference with RMSProp is based on the analysis of the update rule. When we consider the operation x + Δx, we assume that both terms have the same unit; however, the author noticed that the adaptive learning rate η(θ_i) obtained with RMSProp (as well as AdaGrad) is unitless (instead ...