Appendix D. Autodiff
This appendix explains how TensorFlow’s autodifferentiation (autodiff) feature works, and how it compares to other solutions.
Suppose you define a function f(x, y) = x^{2}y + y + 2, and you need its partial derivatives ∂f/∂x and ∂f/∂y, typically to perform Gradient Descent (or some other optimization algorithm). Your main options are manual differentiation, finite difference approximation, forwardmode autodiff, and reversemode autodiff. TensorFlow implements reversemode autodiff, but to understand it, it’s useful to look at the other options first. So let’s go through each of them, starting with manual differentiation.
Manual Differentiation
The first approach to compute derivatives is to pick up a pencil and a piece of paper and use your calculus knowledge to derive the appropriate equation. For the function f(x, y) just defined, it is not too hard; you just need to use five rules:

The derivative of a constant is 0.

The derivative of λx is λ (where λ is a constant).

The derivative of x^{λ} is λx^{λ – 1}, so the derivative of x^{2} is 2x.

The derivative of a sum of functions is the sum of these functions’ derivatives.

The derivative of λ times a function is λ times its derivative.
From these rules, you can derive Equation D1.
Equation D1. Partial derivatives of f(x, y)
$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial f}{\partial x}}& ={\displaystyle \frac{\partial \left({x}^{2}y\right)}{\partial x}}+{\displaystyle \frac{\partial y}{\partial x}}+{\displaystyle \frac{\partial 2}{\partial x}}=y{\displaystyle \frac{\partial \left({x}^{2}\right)}{\partial x}}+0+0=2xy\hfill \\ \hfill {\displaystyle \frac{\partial f}{\partial y}}& ={\displaystyle \frac{\partial \left({x}^{2}y\right)}{\partial y}}+{\displaystyle \frac{\partial y}{\partial y}}+{\displaystyle \frac{\partial 2}{\partial y}}={x}^{2}+1+0={x}^{2}+1\hfill \end{array}$$This approach can ...
Get HandsOn Machine Learning with ScikitLearn, Keras, and TensorFlow, 2nd Edition now with O’Reilly online learning.
O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.