June 2026
Intermediate
384 pages
4h 17m
Japanese
Content preview from ゼロから作るDeep Learning ❻ ―LLM編
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
1章トークナイザ基本編
本章では、LLMの処理の第1段階であるトークナイザの実装に取り組みます。まずは「文字単位」のシンプルなトークナイザを作り、その仕組みを理解することから始めます。そこから段階的に発展させ、最終的にはBPE（Byte Pair Encoding）トークナイザを完成させます。BPEは効率的にトークンを学習できるアルゴリズムであり、現代の多くのLLMで利用されています。
1.1 文字単位のトークン化
トークン化とは、テキストを「トークン」と呼ばれる単位に分割し、モデルが扱える数値（ID）の列に変換することを指します。トークン化には様々な方法がありますが、ここでは最もシンプルな文字単位のトークン化について学びます。
1.1.1 文字をIDに変換する
文字単位のトークン化は、テキストを1文字ごとに分割します。Pythonではlist()関数を使って、テキストを文字のリストに変換できます。
ch01/01_char_tokenizer.py
text="hello世界"
(list(text))# ['h', 'e', 'l', 'l', 'o', '世', '界', '
次のステップは、これらの文字を数値（ID）に変換することです。ここで役立つのが ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.