Deep Learning with PyTorch Quick Start Guide

Learning rate scheduler

The torch.optim.lr_schedular class serves as a wrapper around an to schedule the learning rate according to a specific function multiplied by the initial learning rate. The learning rate scheduler can be applied separately to each parameter group. This can speed up training time since, typically, we are able to use larger learning rates at the beginning of the training cycle and shrink this rate as the optimizer approaches minimal loss. Once a scheduler object is defined, it is typically stepped every epoch using scheduler.step(). There are a number of learning rate scheduler classes available in PyTorch, and the most common one is shown in the following code:

optim.lr_schedular.LambdaLR(optimizer, lr_lambda, last_epoch ...