Apache Spark Deep Learning Cookbook

How to do it...

The following section walks through the steps to apply sentiment score to the dataset.

1. Create a sentiment score function, sentiment_score, using the following script:

from textblob import TextBlobdef sentiment_score(chat):    return TextBlob(chat).sentiment.polarity

2. Apply sentiment_score to each conversation response in the dataframe using the following script:
3. Create a lambda function, called sentiment_score_udf, that maps sentiment_score into a user-defined function within Spark, udf, to each transaction and specifies the output type of FloatType() as seen in the following script:

from pyspark.sql.types import FloatTypesentiment_score_udf = F.udf(lambda x: sentiment_score(x), FloatType())

4. Apply the function, sentiment_score_udf ...