Hands-On Neural Networks with Keras

Thinking on the gradient level

The activation matrix at a given time step is a function of the activation matrix from the previous time step. Hence, we are forced to recursively define the loss at time step three as a product of the sub-gradients of layer weights from previous time steps:

Here, (L) represents the loss, (W) represents the weight matrices of a time step, and the x values are the inputs at a given time steps. Mathematically, this is equivalent to the following:

The derivatives of these functions are stored in a Jacobean matrix, ...