Deep Learning with PyTorch

Learning rate picking strategies

Finding the right learning rate for training the model is an ongoing area of research where a lot of progress has been made. PyTorch provides some of the techniques to tune the learning rate, and they are provided in the torch.optim.lr_sheduler package. We will explore some of the techniques that PyTorch provides to choose the learning rates dynamically:

• StepLR: This scheduler takes two important parameters. One is step size, which denotes for what number of epochs the learning rate has to change, and the second parameter is gamma, which decides how much the learning rate has to be changed.

For a learning rate of 0.01, step size of 10, and gamma size of 0.1, for every 10 epochs the learning rate changes ...