Deep Learning Essentials

Generating training data

In the Skip-Gram model, we generate pairs of words for training as follows:

An example of generating positive training pairs, in which the window covers the beginning of the document

A positive training pair can be generated as follows:

An example of generating positive training pairs

From these illustration, one can easily see that the network is going to learn the statistics from the number of times each pairing (target, context) shows up. For example, the model may see more samples of York, New or New, York rather ...