February 2025
Intermediate
376 pages
9h 25m
Japanese
Content preview from 仕組みからわかる大規模言語モデル 生成AI時代のソフトウェア開発入門
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
ここでは、抽出したテキストのページ数を表示しています。また、0から数えた100 ページ
目のテキストの先頭100文字を表示しています。このプログラムを実行するとPDF ファイル
からテキストが抽出され、ページ数と40 ページ目のテキストが表示されます。手元の環境で
実行した結果を次に示します。
O U T
ページ数: 307
100ページ目： 図表4-1-5-1 企業の研究費の割合（2021年度）
情報通信機械
器具製造業 ,
1兆226億円,
7.2%電気機械器具
製造業 ,
8,377 億円,
5.9%
電子部品・デバイス・
電子回
この結果から、合計307ページのPDF ファイルからテキストが抽出されていることがわか
ります。また、100ページ目には企業の研究費の割合に関する情報が含まれていることが確認
できます。
5.6.2
テキストの分割
次に、このテキストを分割する処理を行っていきます。テキストの分割では、文や段落な
ど、適切な単位にテキストを分割します。LangChainでは、様々なテキスト分割の方法を提供
しています。
表5.6.2 に、LangChain でサポートされているテキスト分割タイプを示します。
表5.6.2
テキスト分割タイプ 説明
CharacterTextSplitter テキストを個々の文字に分割します。
RecursiveCharacterTextSplitter 定義された区切り文字（例: ”\n”, ” ”, ”.”）に基づいてテキストを分割します。 ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.