August 2015
Intermediate to advanced
312 pages
7h 28m
Japanese
Content preview from 初めてのSpark
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
176
9 章 Spark SQL
ることができます。そういったソースの中には、
Hive
のテーブルや、
JSON
や
Parquet
のファイ
ルなどが含まれます。加えて、これらのソースに対して
SQL
でクエリを実行し、一部のフィール
ドだけを選択した場合、
Spark SQL
はソースのフィールドのデータ中の一部だけをうまくスキャ
ンしてくれます。それとは異なり、
SparkContext.hadoopFile
のように単純なソースでは、全ての
データがスキャンされてしまいます。
これらのデータソース以外にも、スキーマを割り当てて通常の
RDD
を
SchemaRDD
に変換す
ることもできます。こうすれば、元々のデータが
Python
や
Java
のオブジェクトであったとして
も、簡単に
SQL
が書けるようになります。いくつもの値を
1
度に計算したい場合には、
SQL
のク
エリの方が簡潔になることは良くあることです（例えば、年齢の平均値と最大値、そしてユーザー
ID
のユニーク数を
1
度の処理で計算したい場合など）。加えて、これらの
RDD
は、他の
Spark
の
SQL
データソースから生成した
SchemaRDD
と結合することも容易です。このセクションでは、
外部のソースと共に、こういった方法で
RDD
を使うやり方も見ていきます。
9.3.1
Apache Hive
データを
Hive
からロードする場合、
Spark SQL
は
Hive
がサポートするすべてのストレー
ジフォーマット（
SerDe
）をサポートします。これには、テキストファイル、
RCFile
、
ORC ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.