August 2015
Intermediate to advanced
312 pages
7h 28m
Japanese
Content preview from 初めてのSpark
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
232
11 章 MLlib を使った機械学習
Word2Vec
Word2Vec
（
https://code.google.com/p/word2vec/
）
†
は、ニューラルネットワークを基盤とす
るテキスト用の特徴抽出アルゴリズムで、データを多くの下流のアルゴリズムに渡すために使うこ
とができます。
Spark
には、
mllib.feature.Word2Vec
クラスに
Word2Vec
の実装があります。
Word2Vec
のトレーニングをするには、
String
（単語ごとに
1
つ）の
Iteratable
として表現さ
れたドキュメントのコーパスを渡します。「
11.4.3 TF-IDF
」と同様に、単語は正規化（すなわちす
べて小文字にした上で、句読点や数字を取り除くといった処理）することが望ましいでしょう。モ
デルをトレーニングすると（
Word2Vec.fit(rdd)
を使います）
Word2VecModel
が返されるので、そ
の
transform()
を使えば、単語をベクトルに変換することができます。注意しなければならないの
は、
Word2Vec
のモデルの大きさは、ボキャブラリー中の単語数に、ベクトルのサイズ（デフォル
トは
100
）をかけたものに等しくなるということです。サイズを制限するために、標準的な辞書に
ない単語はフィルタリングして除外すると良いかもしれません。概して、ボキャブラリーのサイズ
としては
100,000
語が良い大きさです。
11.4.4
計理
基本的な統計処理は、データ分析の重要な部分を占めます。これは、アドホックな探求でも、機 ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.