Frank Kane's Taming Big Data with Apache Spark and Python

More things you can do with DataFrames

As you can see, DataFrame is a very powerful syntax. Let's look at a few more examples. You can call show if you're debugging to show the top 20 rows in your DataFrame:

myResultDataFrame.show() 

You can select a given row and create a new DataFrame out of it:

myResultDataFrame.select("someFieldName") 

Another neat thing is that you can construct little expressions inside these queries. For example, you could do the following:

myResultDataFrame.filter(myResultDataFrame("someFieldName" > 200) 

From the DataFrame, this would filter out anything where the value of a given row's someFieldName column is greater than 200. You can use very intuitive syntactic expressions to actually express complicated logic ...