Querying with SQL
Let's run the same queries, except this time, we will do so using SQL queries against the same DataFrame. Recall that this DataFrame is accessible because we executed the .createOrReplaceTempView
method for swimmers
.
Number of rows
The following is the code snippet to get the number of rows within your DataFrame using SQL:
spark.sql("select count(1) from swimmers").show()
The output is as follows:
Running filter statements using the where Clauses
To run a filter statement using SQL, you can use the where
clause, as noted in the following code snippet:
# Get the id, age where age = 22 in SQL spark.sql("select id, age from swimmers where ...
Get Learning PySpark now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.