Statistics for Machine Learning

Nesterov accelerated gradient - NAG

If a ball rolls down a hill and blindly follows a slope, it is highly unsatisfactory and it should have a notion of where it is going so that it knows to slow down before the hill slopes up again. NAG is a way to give the momentum term this kind of prescience.

While momentum first computes the current gradient (small blue vector) and then takes a big jump in the direction of the updated accumulated gradient (big blue vector), NAG first makes a big jump in the direction of the previous accumulated gradient (brown vector), measures the gradient, and then makes a correction (green vector). This anticipatory update prevents the ball from going too fast and results in increased responsiveness and performance: ...