Deep Learning Essentials

Understanding GloVe

GloVe is an unsupervised learning algorithm for obtaining vector representations (embeddings) for words. It has been seen that with similar training hyperparameters, the embeddings generated using the two methods tend to perform very similarly in downstream NLP tasks.

The differences are that Word2Vec is a predictive model, which learns the embeddings to improve their predictive ability by minimizing the prediction loss, that is, the loss (target word | context words; W). In Word2Vec, it’s been formalized as a feed-forward neural network and uses an optimization approach such as SGD to update the network.

On the other hand, GloVe is essentially a count-based model, where a co-occurrence matrix is first built. Each entry ...