June 2025
Beginner to intermediate
438 pages
9h 17m
English
Content preview from Dataproc Cookbook
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
Chapter 4. Serverless Spark and Ephemeral Dataproc Clusters
Dataproc Serverless Spark is an autoscaling serverless product for Spark that simplifies the execution of Spark applications because the user doesn’t have to think about infrastructure in order to run Spark jobs.
Ephemeral Dataproc clusters, also known as transient clusters, are temporary clusters that run until specific jobs are completed or terminated. Throughout this book, we’ll refer to them as ephemeral clusters. Both serverless and ephemeral Dataproc clusters support running jobs within a VPC or default network.
In this chapter, you will gain a clear understanding of when to use Dataproc Serverless Spark and ephemeral Dataproc clusters. You will also learn:
-
How to submit Spark jobs to Dataproc Serverless
-
How to create and run jobs on ephemeral clusters
-
How to configure a Spark history server
-
How to leverage Spark RAPIDS Accelerator
-
How to price and monitor serverless Spark jobs
Let’s dive in and explore how to scale your Spark workloads efficiently!
Running on Dataproc: Serverless Versus Ephemeral Clusters
Problem
Scenario 1: you’re tasked with running a Spark job on Dataproc, with the following three criteria:
-
You want to avoid managing or customizing the cluster, including hardware selection.
-
There’s no requirement to sequence Spark jobs on the same cluster.
-
The objective is to execute a Spark job, not Hadoop MapReduce jobs or HiveQL scripts.
Scenario 2: you have a Spark job, a Hadoop job, ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.