R Deep Learning Essentials - Second Edition

The optimizer parameter

Our implementation of neural networks used gradient descent. When researchers started creating more complicated multilayer neural network models, they found that they took an extraordinarily long time to train. This is because the basic gradient-descent algorithm with no optimization is not very efficient; it makes small steps towards its goal in each epoch regardless of what occurred in previous epochs. We can compare it with a guessing game: one person has to guess a number in a range and for each guess, they are told to go higher or lower (assuming they do not guess the correct number!). The higher/lower instruction is similar to the derivative value, it indicates the direction we must travel. Now let's say that ...