Hands-On Deep Learning Algorithms with Python

Nesterov accelerated gradient

One problem with momentum is that it might miss out the minimum value. That is, as we move closer toward convergence (the minimum point), the value for momentum will be high. When the value of momentum is high while we are near to attaining convergence, then the momentum actually pushes the gradient step high and it might miss out on the actual minimum value; that is, it might overshoot the minimum value when the momentum is high when we are near to convergence, as shown in the following diagram:

To overcome this, Nesterov introduced a new method called Nesterov accelerated gradient (NAG).

The fundamental motivation ...