Mastering Machine Learning with Spark 2.x

The desc column transformation

The next column we will explore is desc. Our motivation is still to mine any possible information from it and improve model's quality. The desc column contains purely textual descriptions for why the lender wishes to take out a loan. In this case, we are not going to treat them as categorical values since most of them are unique. However, we will apply NLP techniques to extract important information. In contrast to the emp_title column, we will not use the Word2Vec algorithm, but we will try to find words that are distinguishing bad loans from good loans.

For this goal, we will simply decompose descriptions into individual words (that is, tokenization) and assign weights to each used word with the help of tf-idf ...