February 2018
Beginner to intermediate
258 pages
5h 47m
English
If you follow the preceding workflow, and stop from time to time to see the results (which you absolutely should, by the way) you will notice that there is a lot of garbage around—words with upper and lower case, punctuation and so on. What happens if you improve this workflow by properly parsing the words? You can use the tokenizers library instead of the space_tokenizer function from text2vec to remove stopwords and punctuation in a single line:
library(tokenizers) tokens <- tokenize_words(imdb$review, stopwords = stopwords())
The full code is now:
library(plyr)library(dplyr)library(text2vec)library(tidytext)library(caret)imdb <- read.csv("./data/labeledTrainData.tsv" , encoding = "utf-8" , quote = ""Read now
Unlock full access