Deep Learning with PyTorch

Optimizing network architecture

Once we have calculated the loss of our network, we will optimize the weights to reduce the loss and thus improving the accuracy of the algorithm. For the sake of simplicity, let's see these optimizers as black boxes that take loss functions and all the learnable parameters and move them slightly to improve our performances. PyTorch provides most of the commonly used optimizers required in deep learning. If you want to explore what happens inside these optimizers and have a mathematical background, I would strongly recommend some of the following blogs:

• http://colah.github.io/posts/2015-08-Backprop/
• http://ruder.io/deep-learning-optimization-2017/

Some of the optimizers that PyTorch provides are as follows: ...