Deep Learning Quick Reference

Learning word embeddings with counting

Another way to learn word embeddings is by counting. The Global Vectors for Word Representation or GloVe is an algorithm created by Pennington et al. (https://nlp.stanford.edu/projects/glove/).

GloVe works by creating a very large matrix of word co-occurrences. For some corpora, this is essentially a count of the number of times two words occur nearby each other. The algorithm authors weight this count by how close the words are so that words that are close together contribute more to each count. Once this co-occurrence matrix is created, it's decomposed into a smaller space, resulting in a matrix that is words x features big.

Interestingly enough, the results from word2vec and GloVe are very similar ...