Python Deep Learning Cookbook

Experimenting with different optimizers

The most popular and well known optimizer is Stochastic Gradient Descent (SGD). This technique is widely used in other machine learning models as well. SGD is a method to find minima or maxima by iteration. There are many popular variants of SGD that try to speed up convergence and less tuning by using an adaptive learning rate. The following table is an overview of the most commonly used optimizers in deep learning:

Optimizer	Hyperparameters	Comments
SGD	Learning rate, decay	+ Learning rate directly impacts performance (smaller learning rate avoids local minima) - Requires more manual tuning - Slow convergence
AdaGrad	Learning rate, epsilon, decay	+ Adaptive learning rate for all parameters ...