Hands-On Convolutional Neural Networks with TensorFlow

The optimizer and its hyperparameters

As mentioned before, the job of the optimizer is to update the network weights in a way that is going to minimize the training loss error. In all deep learning libraries such as TensorFlow, there is only really one family of optimizer used and that is the gradient descent family of optimizers.

The most basic of these is simply called gradient descent (sometimes called vanilla gradient descent), but more complex ones that try to improve on it have been developed. Some popular ones are:

• Gradient descent with momentum
• RMSProp
• Adam

All of TensorFlow’s different optimizers can be found in the tf.train class. For example the Adam optimizer can be used by calling tf.train.AdamOptimizer().

As you may suspect, ...