August 2015
Intermediate to advanced
312 pages
7h 28m
Japanese
Content preview from 初めてのSpark
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
108
6 章 Spark の高度なプログラミング
ドキュメンテーション（
http://spark.apache.org/docs/latest/api/scala/index.html#package
）
に説明があります。加算の操作としては、単に数値を加えるだけではなく、交換可能かつ結合可能
な操作であれば、任意の操作を使うことができます。例えば、合計値を追跡するための加算の代わ
りに、それまでに現れた最大値を追跡することもできます。
操作
op
が交換可能であるとは、任意の
a
,
b
という値に対して
a op b = b op a
であるということ
です。操作
op
が結合可能であるとは、任意の
a
,
b
,
c
という値に対して
(a op b) op c = a op (b
op c)
であるということです。例えば、Spark のアキュムレータで広く使われる
sum
や
max
は、交
換可能かつ結合可能な操作です。
6.3
ードト数
Spark
における
2
種類目の共有変数であるーは、大きなリードオンリーの変
数をプログラム中で効率的にすべてのワーカーノードに送信し、
Spark
の操作で使ってもらうため
のものです。ブロードキャスト変数は、例えばアプリケーションが大規模なリードオンリーのルッ
クアップ用のテーブルや、あるいは機械学習のアルゴリズムの大規模な特徴ベクトルを全ノードに
送りたいような場合に便利です。
Spark
は、クロージャ中で参照されたすべての変数を、自動的にワーカーノード群に送信するこ ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.