Deep Learning from Scratch

Appendix A. Deep Dives

In this section, we dive deep into a few technical areas that are important to understand for completion, but are not essential.

Matrix Chain Rule

First up is an explanation of why we can substitute W^T for $\frac{\partial \nu }{\partial u}\left(X\right)$ in the chain rule expression from Chapter 1.

Remember that L is literally:

$$\sigma \left(X{W}_{11}\right)+\sigma \left(X{W}_{12}\right)+\sigma \left(X{W}_{21}\right)+\sigma \left(X{W}_{22}\right)+\sigma \left(X{W}_{31}\right)+\sigma \left(X{W}_{32}\right)$$

where this is shorthand for the fact that:

$$\sigma \left(X{W}_{11}\right)=\sigma (x_{11}\times w_{11}+x_{12}\times w_{21}+x_{13}\times w_{31})$$

$$\sigma \left(X{W}_{12}\right)=\sigma (x_{11}\times w_{12}+x_{12}\times w_{22}+x_{13}\times w_{32})$$

and so on. Let’s zoom in on just one of these expressions. What would it look like if we took the partial derivative of, say, $\sigma \left(X{W}_{11}\right)$ with respect to every element of $X$ (which is ultimately what we’ll want to do with all six components of $L$)?

Well, since:

$$\sigma \left(X{W}_{11}\right)=\sigma (x_{11}\times w_{11}+x_{12}\times w_{21}+x_{13}\times w_{31})$$

it isn’t too hard to see that the partial derivative of this with respect to $x_1$, via a very simple application of the chain rule, is:

$$\frac{\partial \sigma }{\partial u}\left(X{W}_{11}\right)\times w_{11}$$

Since the only thing that x₁₁ is multiplied by in the XW₁₁ expression is w₁₁, the partial derivative with respect to everything else is 0.

So, computing the partial derivative of σ(XW₁₁) with respect to all of the elements of X gives us the following overall expression for $\frac{\partial \sigma \left(X{W}_{11}\right)}{\partial X}$:

$$\frac{\partial \sigma \left(X{W}_{11}\right)}{\partial X}=\left[\begin{array}{c}\frac{\partial \sigma }{\partial u}\left(X{W}_{11}\right)\times w_{11}\end{array}\right]$$