Hands-On Natural Language Processing with Python

A comparison of skip-gram and CBOW model architectures

The following diagram shows a comparison of the CBOW and skip-gram model architectures:

This diagram shows how skip-gram learns to predict the target word by using the words nearby. On the other hand, CBOW learns to predict the target word based on the words present in its context, which is represented as a bag-of-words, present in a fixed-size window that is around the target word.

In general, the skip-gram approach tends to produce better word representations when the datasets are larger. Hence, we will concentrate on building a skip-gram model for the rest of this chapter. We will also ...