Feature Engineering Made Easy

The clustering model

We can tokenize the text into sentences so that we expand our dataset. We imported a function called sent_tokenize from the nltk package (natural language toolkit). This function will take in a single string and output the sentence as an ordered list of sentences separated by punctuation. For example:

sent_tokenize("hello! I am Sinan. How are you??? I am fine")['hello!', 'I am Sinan.', 'How are you???', 'I am fine']

We will apply this function to our entire corpus using some reduce logic in Python. Essentially, we are applying the sent_tokenize function to each review and creating a single list called sentences that will hold all of our sentences:

sentences = reduce(lambda x, y:x+y, texts.apply(lambda x: sent_tokenize ...