Advanced Deep Learning with Python

Word2Vec

A lot of research has gone into creating better word embedding models, in particular by omitting learning the probability function over sequences of words. One of the most popular ways to do this is with word2vec (http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf and https://arxiv.org/abs/1301.3781, https://arxiv.org/abs/1310.4546). Similar to NPLM, word2vec creates embedding vectors based on the context (surrounding words) of the word in focus. It comes in two flavors: continuous bag of words (CBOW) and Skip-gram. We'll start with CBOW and then we'll discuss Skip-gram.