Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition

Appendix D. Autodiff

This appendix explains how TensorFlow’s autodifferentiation (autodiff) feature works, and how it compares to other solutions.

Suppose you define a function f(x, y) = x²y + y + 2, and you need its partial derivatives ∂f/∂x and ∂f/∂y, typically to perform Gradient Descent (or some other optimization algorithm). Your main options are manual differentiation, finite difference approximation, forward-mode autodiff, and reverse-mode autodiff. TensorFlow implements reverse-mode autodiff, but to understand it, it’s useful to look at the other options first. So let’s go through each of them, starting with manual differentiation.

Manual Differentiation

The first approach to compute derivatives is to pick up a pencil and a piece of paper and use your calculus knowledge to derive the appropriate equation. For the function f(x, y) just defined, it is not too hard; you just need to use five rules:

• The derivative of a constant is 0.

• The derivative of λx is λ (where λ is a constant).

• The derivative of x^λ is λx^{λ – 1}, so the derivative of x² is 2x.

• The derivative of a sum of functions is the sum of these functions’ derivatives.

• The derivative of λ times a function is λ times its derivative.

From these rules, you can derive Equation D-1.

Equation D-1. Partial derivatives of f(x, y)

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial f}{\partial x}}& ={\displaystyle \frac{\partial \left(x^{2}y\right)}{\partial x}}+{\displaystyle \frac{\partial y}{\partial x}}+{\displaystyle \frac{\partial 2}{\partial x}}=y{\displaystyle \frac{\partial \left(x^2\right)}{\partial x}}+0+0=2xy\hfill \\ \hfill {\displaystyle \frac{\partial f}{\partial y}}& ={\displaystyle \frac{\partial \left(x^{2}y\right)}{\partial y}}+{\displaystyle \frac{\partial y}{\partial y}}+{\displaystyle \frac{\partial 2}{\partial y}}=x^2+1+0=x^2+1\hfill \end{array}$$

This approach can ...