Deep Learning Quick Reference

Teacher forcing

As seen in the illustration above, when predicting an output at some place in the sequence y_t(n), we use y_t(n-1) as the input to the LSTM. We then use the output from this time step to predict y_t(n+1).

The problem with doing this in training is that if y_t(n-1) is wrong, y_t(n) will be even more wrong. This chain of increasing wrongness can make things very very slow to train.

A somewhat obvious solution to this problem is to replace each sequence prediction at each time step with the actual correct sequence at that time step. So, rather than using the LSTM prediction for y_t(n-1), we would use the actual value from the training set.

We can give the model's training process a boost by using this concept, which happens to be called ...