Deep Learning Essentials

Cells in RNN and unrolling

All recurrent neural networks can be thought of as a chain of repeating models/cells in the dimension of time. This repeating module/cell can simply be a single tanh layer. One way to understand this is to unroll, or unravel, the architecture into each time step, and treat each time step as a layer. We can see that the depth of the RNN is essentially decided by the length of the time steps or length of the sequences. The first element of the sequence, such as the word of a sentence, is equivalent to the first layer.

The following figure shows the unrolling of the single recurrent cell in the timeline:

The unrolling ...