Mastering Python for Finance - Second Edition

Optimizers

Optimizers help to tweak the model weights optimally in minimizing the loss function. There are several types of optimizers that you may come across in deep learning:

• AdaGrad (adaptive gradient)
• Adam (adaptive moment estimation)
• LBFGS (limited-memory Broyden-Fletcher-Goldfarb-Shannon)
• Rprop (resilient backpropagation)
• RMSprop (root mean square propagation)
• SGD (stochastic gradient descent)

Adam is a popular choice of optimizer, and is seen as a combination of RMSprop and SGD with momentum. It is an adaptive learning rate optimization algorithm, computing individual learning rates for different parameters.