Apache Spark Deep Learning Cookbook

How to do it...

The following section walks through the steps to profile the text data.

1. Execute the following script to group the label column and to generate a count distribution:

df.groupBy("label") \   .count() \   .orderBy("count", ascending = False) \   .show()

2. Add a new column, word_count, to the dataframe, df, using the following script:

import pyspark.sql.functions as Fdf = df.withColumn('word_count', F.size(F.split(F.col('response_text'),' ')))

3. Aggregate the average word count, avg_word_count, by label using the following script:

df.groupBy('label')\  .agg(F.avg('word_count').alias('avg_word_count'))\  .orderBy('avg_word_count', ascending = False) \  .show()