Deep Learning with PyTorch

Freeze the embedding layer weights

It is a two-step process to tell PyTorch not to change the weights of the embedding layer:

1. Set the requires_grad attribute to False, which instructs PyTorch that it does not need gradients for these weights.
2. Remove the passing of the embedding layer parameters to the optimizer. If this step is not done, then the optimizer throws an error, as it expects all the parameters to have gradients.

The following code demonstrates how easy it is to freeze the embedding layer weights and instruct the optimizer not to use those parameters:

model.embedding.weight.requires_grad = Falseoptimizer = optim.SGD([ param for param in model.parameters() if param.requires_grad == True],lr=0.001)

We generally pass all the model ...