To have a function applied to a group of rows (exactly as in the case of SQL GROUP BY), you can use two similar methods. In the following example, we want to compute the average balance per gender:
In:(df.na.fill({'gender': "U", 'balance': 0.0}) .groupBy("gender").avg('balance').show())Out: +------+------------+ |gender|avg(balance)| +------+------------+ | F| -0.25| | M| 2.0| | U| 7.5| +------+------------+
So far, we've worked with DataFrames, but, as you've seen, the distance between DataFrame methods and SQL commands is minimal. Actually, using Spark, it is possible to register the DataFrame as a SQL table to fully enjoy the power of SQL. The table is saved in memory and distributed in a way ...
