August 2015
Intermediate to advanced
312 pages
7h 28m
Japanese
Content preview from 初めてのSpark
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
90
5 章 データのロードとセーブ
ファイルがクラスタ内の全ノードから利用できるようにはなっていない場合、
Spark
を通さず
にドライバでそのファイルをローカルからロードし、
parallelize
を呼んでその内容をワーカー群
に分配することができます。ただし、このアプローチは低速になることがあるので、ファイルは
HDFS
、
NFS
、
S3
のような共有ファイルシステムに置くことをお勧めします。
5.3.2
Amazon S3
Amazon S3
は、大量のデータの保存先として、ますます使われるようになってきています。
S3
は、コンピュートノードが
Amazon EC2
内にある場合には特に高速ですが、パブリックなイン
ターネットを経由する場合は、パフォーマンスが簡単に低下してしまうこともあります。
Spark
で
S3
にアクセスするには、まず環境変数の
AWS_ACCESS_KEY_ID
と
AWS_SECRET_ACCESS_
KEY
に
S3
のクレデンシャルを設定しなければなりません。これらのクレデンシャルは、
Amazon
Web Services
のコンソールから作成できます。そして、
s3n://
から始まるパスを、
s3n://
bucket/path-within-bucket
という形式で
Spark
のファイル入力メソッドに渡します。他のファイ
ルシステムと同様に、
S3
でも
Spark
は
s3n://bucket/my-files/*.txt
というようなワイルドカー
ドパスをサポートしています。
Amazon
側から
S3
のアクセスパーミッションのエラーが返されたなら、アクセスキーを指定し ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.