In Spark, in order to run any
algorithm you need to read input data
from a data source, then apply your
algorithm in the form of a set of
PySpark transformations and actions (expressed as a DAG), and finally write
your desired output to a target
data source. So, to write algorithms that perform well, it’s important to understand
reading and writing from and to external
data sources.
In the previous chapters, we have explored
interacting with the built-in data sources
(RDDs and DataFrames) in Spark. In this
chapter, we will focus on how Spark
interfaces with external data sources.
As Figure 7-1 shows, Spark can read data from a huge range of external
storage systems like the Linux filesystem,
Amazon S3, HDFS, Hive tables, and
relational databases (such as
Oracle, MySQL, or PostgreSQL) through
its data source interface. This chapter will show you how to read data in and then convert it into RDDs or DataFrames for
further processing. I’ll also show you
how Spark’s data can be written
back to external storage systems
like files, Amazon S3, and JDBC-compliant
databases.
Figure 7-1. Spark external data sources
Relational Databases
Let’s start with relational databases. A relational database is a collection of data items organized as a set ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month, and much more.
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.
Julian F.
Head of Cybersecurity
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.
Addison B.
Field Engineer
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.
Amir M.
Data Platform Tech Lead
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.
