Deep Learning with PyTorch

N-gram representation

We have seen how text can be represented as characters and words. Sometimes it is useful to look at two, three, or more words together. N-grams are groups of words extracted from given text. In an n-gram, n represents the number of words that can be used together. Let's look at an example of what a bigram (n=2) looks like. We used the Python nltk package to generate a bigram for thor_review. The following code block shows the result of the bigram and the code used to generate it:

from nltk import ngramsprint(list(ngrams(thor_review.split(),2)))#Results[('the', 'action'), ('action', 'scenes'), ('scenes', 'were'), ('were', 'top'), ('top', 'notch'), ('notch', 'in'), ('in', 'this'), ('this', 'movie.'), ('movie.', 'Thor'), ...