Hands-On Artificial Intelligence for Beginners

GRUs

GRUs were developed in 2014 as a new take on the classic LSTM cell. Instead of having four separate gates, it combines the forget and input gates into a single update gate; you can think of this in the sense of: whatever is not written, gets forgotten. It also merges the cell state c_t from the LSTM with the overall hidden state h_t. At the end of a cycle, the GRU exposes the entire hidden state: 

• Update gate: The update gate combines the forget and input gates, essentially stating that whatever is not written into memory is forgotten. The update gate takes in our input data x and multiplies it by the hidden state. The result of that expression is then multiplied by the weight matrix and fed through a sigmoid function: 

• Reset gate ...