June 2026
Intermediate
384 pages
4h 17m
Japanese
Content preview from ゼロから作るDeep Learning ❻ ―LLM編
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
ここから、いよいよCodeBotを作る旅が始まります。CodeBotは、Pythonのコードを書くことのできる小さな言語モデルです。小さくても、対話ができ、指示に従い、計算もこなすことができます。完成時には次のような会話ができるようになります。
You: Hello
Bot: Hello. What do you need?
You: Who are you?
Bot: I'm CodeBot. How can I assist you today?
You: write loop
Bot:
for i in range(10):
if i == 5:
break
You: 7+8=
Bot: 15
I.1 開発の流れ
CodeBotの開発は、次の3つのステップで進みます。
| 章 | テーマ | 内容 |
|---|---|---|
| 1章 | トークナイザ（基本編） | BPEを実装し、テキストをトークンのID列に変換 |
| 2章 | モデル（基本編） | Transformerの基本原理を学び、小型のGPT-2を実装 |
| 3章 | 学習（基本編） | 事前学習、SFT、強化学習でLLMが賢くなる過程を体験 |
これら3つのステップを通して、LLMの核心技術をひととおり体験します。特に「3章 学習基本編」では、モデルが段階的に賢くなっていく様子を実感できるでしょう。ここでは、その学習の流れを簡単に見ていきます。
I.2 事前学習
事前学習では、大量のPythonコードを使って、モデルにプログラミングの基礎を学ばせます。
使用するデータセットはcodebot/tiny_codes.txt ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.