August 2015
Intermediate to advanced
312 pages
7h 28m
Japanese
Content preview from 初めてのSpark
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
259
A.6 バージョン 1.4.0 の主な変更点
•
よりスループットの高いストレージやネットワークが普及したこと
• Spark
の
I/O
まわりの改善が着実に実施されたこと
•
データフォーマットが改善されたこと
•
シリアライゼーションやハッシュ計算が
CPU
バウンドであること
また
Project Tungsten
では具体的に次の方針に則って、パフォーマンスを改善するとしています。
•
明示的にメモリを管理する
• JVM
のオブジェクトでデータを保持することに伴うオーバヘッドの解消
• GC
オーバーヘッドを減らす
•
メモリ内のデータ構造を考慮してデータアクセスする
•
モダンなコンパイラや
CPU
に対応する
•
バイナリデータに対して直接アクセスする
旧来の
Hadoop MapReduce
フレームワークからの改善という当初の目論見は達成されつつあり
ますが、まだまだ進歩は続くことでしょう。
A.6.3
DataFrame API and Spark SQL
• SPARK-2883:
［機能改善］
ORCFile
フォーマットへの対応
• SPARK-2213:
［機能改善］大きなサイズの
JOIN
を最適するためのソートマージ
JOIN
の
追加
• SPARK-5100:
［機能改善］
SQL JDBC server
の
UI
追加
• SPARK-6829:
［機能改善］
DataFrame
が数学的関数に対応
• SPARK-8299:
［機能改善］
DataFrame and SQL
のエラーレポートの改善
• SPARK-1442: ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.