Frank Kane's Taming Big Data with Apache Spark and Python

Using SQL-style functions instead of queries

You don't have to use SQL queries anymore; it's arguably a little bit more efficient to use SQL-style functions instead. Let's discuss a way to do this. Remember in the previous chapters we had to jump through some hoops to reduce things together and actually do things, such as count the number of people by identifying how many people of each age exist, for example? We can do all of this with just one line of code using DataFrames:

# We can also use functions instead of SQL queries: 
schemaPeople.groupBy("age").count().orderBy("age").show() 

Let's take a look at what we're doing here. We're using the cached schemaPeople DataFrame and calling groupBy, using the age column, and then doing a count. ...