February 2025
Intermediate
376 pages
9h 25m
Japanese
Content preview from 仕組みからわかる大規模言語モデル 生成AI時代のソフトウェア開発入門
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
3
テキスト抽出
まず、外部知識となる文書を選択し、それをテキストデータに変換します。文書の形式とし
てはWord、PDF、Markdown、HTML など様々なものが考えられます。これらの文書に含ま
れるテキストデータを抽出し、前処理を行います。前処理には、ヘッダ、フッタ、広告の除去
など不要なデータの削除やテキスト中の表記の統一などが含まれます。
分割
次に、テキストデータを適切なサイズのチャンクに分割します。テキストを分割するのは、
後でLLM に入力する際の制限を考慮してのことです。LLM には最大トークン数を超える入力
はできないため、適切なサイズに分割する必要があります。このため、利用するLLM に適し
た長さにテキストを分割します。チャンクの単位は、文章、段落、固定長のトークン数などが
考えられます。
ベクトル化
次に、分割した個々のチャンクを埋め込みベクトルに変換します。この埋め込みベクトル
は、文書の特徴をベクトル化したものです。埋め込みベクトルの作成にも1.3 で説明した
Transformerがしばしば用いられます。文書をTransformerに入力し、最後のトークンの埋め
込みベクトルを取得します。最後のトークンには、文書全体の情報が集約されているため、文
書全体の特徴を表す埋め込みベクトルとして利用できます。
保存
インデックスの作成の最後の手順は、埋め込みベクトルをデータベースに保存することで
す。ここで利用するデータベースは、埋め込みベクトルを用いた高速な類似度検索に対応可能
なものです。例えば ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.