Hands-On Artificial Intelligence for Beginners

Utilizing the Adam optimizer in our MNIST example

The adaptive gradient descent method (Adam) takes an initial learning rate and adaptively computes updates to it. Adam stores an exponentially decaying average of past squared gradients and of past gradients, which amounts to measuring something similar to momentum. This helps us prevent overshooting or undershooting during our training process.

Adam is easily implemented in TensorFlow with the following command line: 

tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(loss_func)

The tf.train class contains various different optimizers that are executed at runtime and contain TensorFlow's version of the Adam optimizer; it takes our initially defined learning_rate as a parameter. ...