O'Reilly logo

Apache Spark for Data Science Cookbook by Padma Priya Chitturi

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Creating Pandas DataFrames over Spark

A DataFrame is a distributed collection of data organized into named columns. It is equivalent to a table in a relational database or a DataFrame in R/Python Python with rich optimizations. These can be constructed from a wide variety of sources, such as structured data files (JSON and parquet files), Hive tables, external databases, or from existing RDDs.

PySpark is the Python API for Apache Spark which is designed to scale to huge amounts of data. This recipe shows how to make use of Pandas over Spark.

Getting ready

To step through this recipe, you will need a running Spark cluster either in pseudo distributed mode or in one of the distributed modes, that is, standalone, YARN, or Mesos. Also, have Python and ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required