August 2015
Intermediate to advanced
312 pages
7h 28m
Japanese
Content preview from 初めてのSpark
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
117
6.6 数値の RDD の操作
最初の方法では、コマンドの呼び出しを、一連のポジション引数として渡しています（コマンド
そのものは、オフセット
0
の引数になります）。
2
番目の方法で渡しているのは
1
つのコマンド文
字列で、
Spark
がそれらをポジション引数に分割することになります。
必要な場合は、
pipe()
でシェルの環境変数を指定することもできます。単純に環境変数の
map
を
pipe()
の
2
番目の引数として渡せば、
Spark
がそれらを設定してくれます。
以上で、
RDD
の要素を外部のコマンドを通じて処理するために
pipe()
を使う方法と、そういっ
たコマンドスクリプトを、クラスタの各ノードが見つけられるように配布する方法について理解で
きました。
6.6
数値の
RDD
の
Spark
には、数値データを含む
RDD
に対する記述統計の操作群があります。これらの操作と、
さらに複雑な統計や機械学習のメソッドについては、後ほど
11
で説明します。
Spark
の数値の操作はストリーミングアルゴリズムを利用して実装されているので、モデルの構
築を要素
1
つずつ行っていくことができます。記述統計は、データに対する
1
度のパスだけです
べて計算され、
stats()
が呼ばれると、
StatsCounter
オブジェクトとして返されます。
6-2
は、
StatsCounter
オブジェクトで利用できるメソッドのリストです。
6-2
StatsCounter
count()
RDD
内の要素数
mean()
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.