O'Reilly logo

Learning PySpark by Denny Lee, Tomasz Drabas

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Data operations

We have already presented some of the most common methods you will use with DataShapes (for example, .peek()), and ways to filter the data based on the column value. Blaze has implemented many methods that make working with any data extremely easy.

In this section, we will review a host of other commonly used ways of working with data and methods associated with them. For those of you coming from pandas and/or SQL, we will provide a respective syntax where equivalents exist.

Accessing columns

There are two ways of accessing columns: you can get a single column at a time by accessing them as if they were a DataShape attribute:

traffic.Year.head(2)

The preceding script produces the following output:

You can also use indexing that allows ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required