Running Spark in YARN
In the previous section, we covered Spark working in a Standalone cluster, and while on-premises deployment on Standalone clusters accounts for 42% of all Spark deployments, Spark deployment on YARN accounts for 36% of all on-premises deployments. These figures were taken from the Apache Spark survey report published by data bricks in June of 2016.
Figure 8.15: YARN deployments - Databricks survey report: Page 10 (Databricks.com)
YARN has been supported since 0.6 release of Apache Spark. In order to run your application on YARN you need to ensure you have the correct configuration files for your Hadoop cluster, and the following ...
Get Learning Apache Spark 2 now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
