Python Deep Learning - Second Edition

word2vec

A lot of research has gone into creating better word embedding models, in particular by omitting learning the probability function over sequences of words. One of the most popular ways to do this is via word2vec (http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf and https://arxiv.org/pdf/1301.3781.pdf). To create embedding vectors with a word2vec model, we'll need a simple neural network that has the following:

• It has an input, hidden, and an output layer
• The input is a single one-hot encoded word representation
• The output is a single softmax, which predicts the most likely word to be found in the context (proximity) of the input word

We'll train the network with gradient ...