August 2015
Intermediate to advanced
312 pages
7h 28m
Japanese
Content preview from 初めてのSpark
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
47
4
ー値 アの 理
本章は、キー／値ペアの
RDD
の扱い方を取り上げます。キー／値ペアの
RDD
は、
Spark
での
多くの操作に必要となる、広く使われるデータ型です。キー／値ペアの
RDD
は集計を行う場合に
使われることが多く、データをキー／値の形式にするために、多少の
ETL
（抽出、変換、ロード）
処理を最初に行わなければならないことがしばしばあります。キー／値ペアの
RDD
には、新しい
操作があります（例えば、各製品ごとのレビュー数のカウント、同じキーを持つデータのグループ
化、
2
つの別々の
RDD
のグループ化など）。
また、ノード間でのペア
RDD
の配置を制御する高度な機能である、ーについ
ても議論します。パーティショニングを制御して、データが同じノード内でまとめてアクセスされ
るようにすることによって、アプリケーションの通信コストを劇的に下げられる場合もあります。
そうなれば、処理速度は大きく向上します。パーティショニングの説明では、例として
PageRank
を取り上げます。分散データセットにおける適切なパーティショニングの選択は、ローカルのデー
タにおける適切なデータ構造の選択に似ています。すなわちどちらの場合も、データのレイアウト
が大きくパフォーマンスに影響するのです。
4.1
ア
RDD
理
Spark
では、キー／値ペアを含む
RDD
に対して、特別な操作群が提供されています。キー／値
ペアを含む
RDD
は、ペア
RDD
と呼ばれます。ペア
RDD
は、それぞれのキーに対して並列に処 ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.