August 2015
Intermediate to advanced
312 pages
7h 28m
Japanese
Content preview from 初めてのSpark
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
65
4.5 データのパーティショニング（上級編）
に
persist()
を追加しておくべきです。そうすべき理由は、これまでの例で
userData
に対して
persist()
が必要だったのと同じ理由です。
persist()
をしておかなければ、それ以降の
RDD
の
アクションは、
pertitioned
の系統全体を評価し直すことになるので、
pairs
に対して何度もハッ
シュパーティショニングが行われることになってしまいます。
4.5.2
ーテニングのる
Spark
の操作の多くは、ネットワーク越しにキーに基づくシャッフルを起こします。それらは
すべて、パーティショニングの恩恵を受けることができます。
Spark 1.0
では、パーティショニ
ングの恩恵を受けることができる操作は、
cogroup()
、
groupWith()
、
join()
、
leftOuterJoin()
、
rightOuterJoin()
、
groupByKey()
、
reduceByKey()
、
combineByKey()
、
lookup()
です。
reduceByKey()
のように、単一の
RDD
に対して働く操作をパーティション化された
RDD
で動
作させると、各キーのすべての値は、単一のマシン上でーに計算され、ローカルに
reduce
された最終の値だけが、各ワーカーノードからマスターに戻されます。
cogroup()
や
join()
のよ
うに
2
つの
RDD
に対する操作の場合、事前にパーティショニングが行われていれば、最低でも
RDD
の
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.