Building Machine Learning Systems with Python - Third Edition

Recurrent neural networks

All the networks we saw earlier have one layer feeding data to another layer, and there was no loop. Recurrent networks loop on themselves, so what happens is that the new value of an output also depends on the past internal state of a node as well as its input. This can be summed up in the following picture:

Theoretically, these networks can be trained, but it is a hard task, especially in text prediction when a new word may depend on other words that are long gone (think of clouds up in the sky where the predicted word sky depends on cloud that is three words in the past).

More information on this problem can be ...