Deep Learning Essentials

The hidden layer

The hidden layer, in this case, does not have any activation functions. The connection between the input layer and the hidden layer can be thought of as a weight matrix, W_{V X N} , where N is the number of the neurons in the hidden layer. W_{V X N} is with V rows, that is one for every word in the vocabulary, and N columns, that is one for every hidden neuron. The number N will be the embedding vector length. There is another auxiliary matrix, W′_{N X V}, which connects the hidden layer and the output layer, and the similarity of the word, W, with the hallucinated context word (out-of-window word) is minimized.