Hands-On Deep Learning with Go

Momentum

When thinking about optimization of gradient descent, we can certainly use intuition from real life to help to inform our methods. One example of this is momentum. If we imagine that most error gradients are really like a bowl, with the desired point in the middle, if we start from the highest point of the bowl, it could take us a long time to get to the bottom of the bowl.

If we think about some real-life physics, the steeper the side of the bowl, the quicker a ball would fall along the side as it gained momentum. Taking this as inspiration, we get what we can consider the momentum variation of SGD; we try to help to accelerate the descent down the gradient by considering that, if the gradient continues to go down the same direction, ...