Mastering Machine Learning with Spark 2.x

Feature extraction

At this stage, we have only a raw text representing reviews, which is not sufficient to run any machine learning algorithm. We need to transform the text into a numeric format, aka perform the so-called "feature extraction" (it is as it sounds; we are taking the input data and extracting features which we will use to train a model). The method generates some new features based on input feature(s). There are many methods regarding how the text can be transformed into numeric features. We can count the number of words, length of text, or number of punctuations. However, to represent text in a systematic way that would reflect a text structure, we need more elaborate methods.