Neural Network Projects with Python

Input gate

The next gate is the Input gate (i). The Input gate (i) controls how much information to pass to the current cell state. The input gate of an LSTM is highlighted in the following diagram:

Just like the forget gate, the Input gate (i) takes as input the concatenation of the Previous Hidden State (h_t-1) and the current Input (x_t). It then passes two copies of the concatenated vector through a sigmoid function and a tanh function, before multiplying them together.

The output of the input gate, i, is as follows:

At this point, we have ...