Grouping and creating tables in-memory

To have a function applied to a group of rows (exactly as in the case of SQL GROUP BY), you can use two similar methods. In the following example, we want to compute the average balance per gender:

In:(df.na.fill({'gender': "U", 'balance': 0.0})    .groupBy("gender").avg('balance').show())Out: +------+------------+      |gender|avg(balance)|      +------+------------+      |     F|       -0.25|      |     M|         2.0|      |     U|         7.5|     +------+------------+

So far, we've worked with DataFrames, but, as you've seen, the distance between DataFrame methods and SQL commands is minimal. Actually, using Spark, it is possible to register the DataFrame as a SQL table to fully enjoy the power of SQL. The table is saved in memory and distributed in a way ...

Get Python Data Science Essentials - Third Edition now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.