Hands-On Deep Learning with Go

Vanilla RNNs

According to their more utopian description, RNNs are able to do something that the networks we've covered so far cannot: remember. More precisely, in a simple network with a single hidden layer, the network's output, as well as the state of that hidden layer, are combined with the next element in a training sequence to form the input for a new network (with its own trainable, hidden state). A vanilla RNN can be visualized as follows:

Let's unpack this a bit. The two networks in the preceding diagram are two different representations of the same thing. One is in a Rolled state, which is simply an abstract representation of the ...