Deep Learning with Python: Learn Best Practices of Deep Learning Models with PyTorch

© Nikhil Ketkar, Jojo Moolayil 2021

N. Ketkar, J. MoolayilDeep Learning with Pythonhttps://doi.org/10.1007/978-1-4842-5364-9_4

4. Automatic Differentiation in Deep Learning

Nikhil Ketkar¹   and Jojo Moolayil²

(1)

Bangalore, Karnataka, India

(2)

Vancouver, BC, Canada

 

While exploring stochastic gradient descent in Chapter 3, we treated the computation of gradients of the loss function 𝛻_xL(x) as a black box. In this chapter, we open the black box and cover the theory and practice of automatic differentiation, as well as explore PyTorch’s Autograd module that implements the same. Automatic differentiation is a mature method that allows for the effortless and efficient computation of gradients of arbitrarily complicated loss functions. This is critical when ...