Deep Learning Essentials

The output layer

The output layer is a softmax regression classifier. It takes a vector of arbitrary real-valued scores (z) and squashes it to a vector of values between zero and one that sums to one:

The output from the hidden layer (the word vector for the input target word, w) is multiplied with the auxiliary matrix W′_{N X V} where each column of it represents a word (let’s assume a word, c). The dot product produces a value which, after normalization, represents the probability of having context word, c, given the input target word, w.

Here’s an illustration of calculating the output of the output neuron for the word car and applying the ...