August 2015
Intermediate to advanced
312 pages
7h 28m
Japanese
Content preview from 初めてのSpark
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
188
9 章 Spark SQL
することになるので、オーバーヘッドが生じてしまうことがあります
†
。現時点ではコード生成は
まだ
experimental
ですが、大きなクエリを使用する処理や、同じクエリを何度も繰り返し実行す
るような処理を行う場合は、使ってみることをお勧めします。
チューニングに使うべきかもしれない
2
つめのオプションは、
spark.sql.inMemoryColumnar
Storage.batchSize
です。
SchemaRDD
をキャッシュする場合、
Spark SQL
はその
RDD
内のレ
コード群を、このオプション（デフォルトは
1000
）で指定されたサイズをバッチとしてグループ
化し、そのバッチ単位で圧縮を行います。このサイズを非常に小さくすれば圧縮率が下がってしま
いますが、一方で非常に大きく指定しても、メモリ内でそれぞれのバッチの構築ができなくなって
しまうかもしれないので、問題になります。テーブル内の行が大きいなら（すなわちフィールド
数が数百に及んだり、
Web
ページのように非常に長くなりうる文字列フィールドが含まれている
場合）、
out-of-memory
エラーが生じないようにバッチサイズを小さくする必要があるかもしれま
せん。行がそれほど大きくないなら、おそらくはデフォルトのバッチサイズは妥当です。これは、
1,000
レコードを超えると、それ以上は圧縮の効果が向上しなくなっていくためです。
9.7
め
Spark SQL
を使い、
Spark
で構造化および半構造化データを扱う方法を見てきました ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.