The forget gate, 
, is responsible for deciding what information should be removed from the cell state (memory). Consider the following sentence:
Harry is a good singer. He lives in New York. Zayn is also a good singer.
As soon as we start talking about Zayn, the network will understand that the subject has been changed from Harry to Zayn, and the information about Harry is no longer required. Now, the forget gate will remove/forget information about Harry from the cell state.
The forget gate is controlled by a sigmoid function. At time step , we pass input , and the previous hidden state, , to the forget gate. It return 0 if the ...
