August 2015
Intermediate to advanced
312 pages
7h 28m
Japanese
Content preview from 初めてのSpark
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
139
7.6 クラスタマネージャ
で動作し、クラスタモードでは、ドライバも
YARN
のコンテナ内で動作することになります。モー
ドの設定は、
spark-submit
の引数の
--deploy-mode
で行います。
Spark
のインタラクティブシェルと
pyspark
は、どちらも
YARN
上で動作させることもできま
す。
HADOOP_CONF_DIR
を設定し、
--master yarn
をこれらのアプリケーションに渡すだけです。こ
れらはユーザーからの入力を受け付けなければならないので、クライアントモードでのみ動作する
ことに注意してください。
使用リソースの設定
YARN
上で動作させる場合、
Spark
のアプリケーションは決められた数のエクゼキュータを使い
ます。この数は、
spark-submit
や
spark-shell
などの
--num-executors
フラグで設定できます。
デフォルトの値は
2
にすぎないので、増やしておきましょう。各エクゼキュータが使用するメモリ
は
--executor-memory
で、
YARN
に要求するコア数は
--executor-cores
で設定することができま
す。ハードウェアリソースが同じであれば、通常
Spark
は、少数の大きなエクゼキュータ（複数
のコアと大量のメモリが割り当てられたエクゼキュータ）の下の方がうまく動作します。これは、
Spark
が各エクゼキュータ間の通信を最適化できることによります。ただし注意が必要なのは、
クラスタ側の設定によってはエクゼキュータの最大サイズに制約があり（デフォルトでは ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.