R Deep Learning Projects

The more, the merrier – calculating n-grams instead of single words

The reason for such an odd result in the previous section can be traced back to the context.  Notice that review 3 had the phrase No nonsense, no gimmicks, which is largely positive, but has two negative words attached to it. How can we take context into account? Enter n-grams. An n-gram is a sequence of n consecutive items (words, or in the case of speech, phonemes) from a given sequence of text or speech. Let's make this clear with an example and use 2-grams, or bigrams:

text_df <- data_frame(line = 1:4, text = text)text_df <- text_df %>% unnest_tokens(bigram, text, token="ngrams", n=2)text_df

This gives us the following:

# A tibble: 70 x 2 line bigram <int> <chr> 1 1 the ...