Apache Spark Deep Learning Cookbook

How to do it...

This section walks through the steps to remove stop words.

1. Execute the following script to extract each word in chat into a string within an array:

df = df.withColumn('words',F.split(F.col('chat'),' '))

2. Assign a list of common words to a variable, stop_words, that will be considered stop words using the following script:

stop_words = ['i','me','my','myself','we','our','ours','ourselves','you','your','yours','yourself','yourselves','he','him','his','himself','she','her','hers','herself','it','its','itself','they','them','their','theirs','themselves','what','which','who','whom','this','that','these','those','am','is','are','was','were','be','been','being','have','has','had','having','do','does','did','doing','a','an','the','and','but','if','or','because','as','until','while', ...