TensorFlow 1.x Deep Learning Cookbook

There's more...

It is often recommended that you start with a high value of learning rate and reduce it as the learning progresses. This helps in fine-tuning the training. We can use the TensorFlow tf.train.exponential_decay method to achieve this. According to the TensorFlow docs:

When training a model, it is often recommended to lower the learning rate as the training progresses. This function applies an exponential decay function to a provided initial learning rate. It requires a global_step value to compute the decayed learning rate. You can just pass a TensorFlow variable that you increment at each training step.The function returns the decayed learning rate. Args: learning_rate: A scalar float32 or float64 Tensor or a Python number. ...