August 2015
Intermediate to advanced
312 pages
7h 28m
Japanese
Content preview from 初めてのSpark
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
28
3 章 RDD を使ったプログラミング
して、もう
1
つの変換として
union()
を使い、
error
もしくは
warning
を含む行の数を出力してみ
ましょう。
3-14
は
Python
で書かれていますが、
union()
関数は、
3
つのどの言語でも同じ
です。
3-14
Python
union()
errorsRDD = inputRDD.filter(lambda x: "error" in x)
warningsRDD = inputRDD.filter(lambda x: "warning" in x)
badLinesRDD = errorsRDD.union(warningsRDD)
union()
は、
1
つではなく
2
つの
RDD
に対して操作を行うという点で、
filter()
とは異なって
います。実際には、変換は任意の数の入力
RDD
に対して操作を行うことができます。
リスト 3-14 と同じ結果を得たい場合、実際には単に
inputRDD
に対し、
error
もしくは
warning
を
検索するフィルタをかける方が良いでしょう。
最終的には、変換によって新しい
RDD
を導出していくたびに、
Spark
はそれぞれの
RDD
間の
依存関係を保持していきます。この依存関係をと呼びます。
Spark
は、必要に応じて
それぞれの
RDD
の計算を行ったり、永続化された
RDD
の一部が失われた場合にそのデータを回
復させたりするために、この情報を利用します。
3-1
は、
3-14
の系統グラフを示してい ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.