Machine Learning with Spark - Second Edition

Skip-gram model

The training objective of the skip-gram model is to find word representations useful for predicting the surrounding words in a document or a sentence. Given a sequence of words w1, w2, w3, . . , wT, skip-gram model maximizes the average log probability shown as following:

c is the size of the training context (which can be a function of the center word wt). Larger c results in more training examples leading to a higher accuracy, at the expense of the training time. The basic skip-gram formulation defines p(wt+j |wt) using the softmax function:

v_w, v' and, w are the input and output vector representations of w, and W is the ...