Hands-On Machine Learning with Scikit-Learn and PyTorch

Appendix A. Autodiff

This appendix explains how PyTorch’s automatic differentiation (autodiff) feature works, and how it compares to other solutions.

Suppose you define a function f(x, y) = x²y + y + 2, and you need its partial derivatives ∂f/∂x and ∂f/∂y, typically to perform gradient descent (or some other optimization algorithm). Your main options are manual differentiation, finite difference approximation, forward-mode autodiff, and reverse-mode autodiff. PyTorch implements reverse-mode autodiff, but to fully understand it, it’s useful to look at the other options first. So let’s go through each of them, starting with manual differentiation.

Manual Differentiation

The first approach to compute derivatives is to pick up a pencil and a piece of paper and use your calculus knowledge to derive the appropriate equation. For the function f(x, y) just defined, it is not too hard; you just need to use five rules:

• The derivative of a constant is 0.

• The derivative of λx is λ (where λ is a constant).

• The derivative of x^λ is λx^{λ – 1}, so the derivative of x² is 2x.

• The derivative of a sum of functions is the sum of these functions’ derivatives.

• The derivative of λ times a function is λ times its derivative.

From these rules, you can derive Equation A-1.

Equation A-1. Partial derivatives of f(x, y)

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial f}{\partial x}}& ={\displaystyle \frac{\partial \left(x^{2}y\right)}{\partial x}}+{\displaystyle \frac{\partial y}{\partial x}}+{\displaystyle \frac{\partial 2}{\partial x}}=y{\displaystyle \frac{\partial \left(x^2\right)}{\partial x}}+0+0=2xy\hfill \\ \hfill {\displaystyle \frac{\partial f}{\partial y}}& ={\displaystyle \frac{\partial \left(x^{2}y\right)}{\partial y}}+{\displaystyle \frac{\partial y}{\partial y}}+{\displaystyle \frac{\partial 2}{\partial y}}=x^2+1+0=x^2+1\hfill \end{array}$$

This approach ...