6
Performance Tuning with Apache Spark
Apache Spark is a powerful and versatile framework for large-scale data processing. It offers high-level APIs in Scala, Java, Python, and R, as well as low-level access to the Spark core engine. Spark supports a variety of workloads, such as batch processing, streaming, machine learning, graph analytics, and SQL queries. However, to get the most out of Spark, you need to know how to optimize its performance and avoid common pitfalls.
In this chapter, you will learn how to performance-tune Apache Spark applications.
We will cover the following recipes in this chapter:
- Monitoring Spark jobs in the Spark UI
- Using broadcast variables
- Optimizing Spark jobs by minimizing data shuffling
- Avoiding data skew
- Caching ...
Get Data Engineering with Databricks Cookbook now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
