Machine Learning for Developers

Training method — backpropagation through time

After the considerable number of model types we've been studying, it's possible that you can already see a pattern in the implementation of the training steps.

For recurrent neural networks, the most well-known error minimization technique is a variation of the well-known (for us) backpropagation methods, with a simple element – backpropagation through time (BPTT) works by unrolling all input timesteps. Each timestep has one input timestep, one copy of the whole network, and one output. Errors are calculated and accumulated for each timestep, and finally the network is rolled back up and the weights are updated.

Spatially, each timestep of the unrolled recurrent neural network can be seen as ...