Dataproc Cookbook

Chapter 11. Performance Tuning and Cost Optimization

In the landscape of big data engineering, optimizing both performance and costs is important. This requires a deep understanding of and well-designed big data applications as well as careful benchmarking to ensure optimal performance. Profiling and benchmarking are usually carried out during or after the application-development phase, which helps to optimize both performance and cost.

This chapter delves into the crucial aspects of performance tuning and cost optimization within Dataproc. You’ll learn how to size Dataproc clusters, benchmark them, choose appropriate disks, utilize Spark and YARN UIs, optimize Spark jobs, profile them using Sparklens, identify errors, and calculate and optimize the cost of your Dataproc clusters.

Sizing a Dataproc Cluster

Problem

Your team is onboarding a new application in Dataproc and wants to estimate the size of the cluster.

Solution

Create a list to capture information about the storage and compute requirements and plan the cluster accordingly.

Discussion

To effectively size a Dataproc cluster, you should choose the right set of components in the cluster, including the following:

• Nature of the cluster (static, ephemeral, or serverless)

• Master and worker machine type

• Number of primary workers

• Number and type of secondary workers

• Disk type and size to be attached to each worker node

• Autoscaling configuration

To choose the proper components, start by asking the following ...