One problem with the Adam algorithm is that it sometimes fails to attain optimal convergence, or it reaches a suboptimal solution. It has been noted that, in some settings, Adam fails to attain convergence or reach the suboptimal solution instead of a global optimal solution. This is due to exponentially moving the averages of gradients. Remember when we used the exponential moving averages of gradients in Adam to avoid the problem of learning rate decay?
However, the problem is that since we are taking an exponential moving average of gradients, we miss out information about the gradients that occur infrequently.
To resolve this issue, the authors of AMSGrad made a small change to the Adam algorithm. ...
