Machine Learning for Developers

The gate and multiplier operation

LSTMs have two fundamental values: remembering important things from the present, and slowly forgetting unimportant things from the past. What kind of mechanism can we use to apply this kind of filtering? It's called the gate operation.

The gate operation basically has a multivariate vector input and a filter vector, which will be dot multiplied with the input, allowing or rejecting the elements from the inputs to be transferred. How do we adjust this gate's filters? This multivariate control vector (marked with an arrow on the diagram) is connected with a neural network layer with a sigmoid activation function. If we apply the control vector and pass through the sigmoid function, we will get a binary-like ...