August 2015
Intermediate to advanced
312 pages
7h 28m
Japanese
Content preview from 初めてのSpark
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
245
11.6 パイプライン API
るので、自分のプログラム中で再利用することがないのであれば、
Python
の
RDD
をキャッシュ
する必要はありません。しかし、
Scala
と
Java
では、キャッシュはユーザー側でやらなければな
らないことなのです。
11.5.4
ーの
特徴ベクトルに含まれている値がほとんどゼロばかりなのであれば、特徴ベクトルを疎なフォー
マットで保存することで、大規模なデータセットの場合、時間と領域を大きく節約できます。領域
については、ゼロではないエントリが最大でも
3
分の
2
程度であれば、
MLlib
の疎な表現は、密
な表現よりも小さくなります。処理のコストについては、ゼロではないエントリが最大で
10%
程
度であれば、概して疎なベクトルの方が低くなります（これは、密なベクトルに比べると、疎な表
現の方がベクトルの要素あたりで多く命令の処理を必要とするためです）。ただし、疎な表現を取
ることで、ベクトルがメモリ中にキャッシュできるかどうかが変わるのであれば、ここで示したよ
りも密なデータに対しても、疎な表現を使うことを検討してみるべきです。
11.5.5
列度
ほとんどのアルゴリズムでは、入力
RDD
のパーティション数を最低でもクラスタ内のコア数
と同等にして、並列性をできる限り活用するべきです。
Spark
は、デフォルトでファイルの各
ブロックごとにパーティションを生成することを思い出してください。このブロックは、通常
は
64MB
です。例えば
sc.textFile("data.txt", ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.