February 2025
Intermediate
376 pages
9h 25m
Japanese
Content preview from 仕組みからわかる大規模言語モデル 生成AI時代のソフトウェア開発入門
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
KLダイバージェンスは非負の値を取り、二つの分布が完全に一致する場合のみ0 になりま
す。値が大きいほど、二つの分布の違いが大きいことを示します。
目的関数におけるKLダイバージェンス
目的関数
J
(
z
)の第 2項では、この KLダイバージェンスにマイナスをかけることで、現在の
方策が教師あり学習済みモデル、すなわちRLHF 学習前の方策から大きく乖離することを防
いでいます。これにより、言語モデルとしての基本的な能力を保持しながら、人間の好みに適
合するように最適化を行うことができます。
b
は、この第2項の重みを制御するハイパーパラメータです。
b
の値を大きくすると、教師
あり学習済みモデルの方策からの乖離に対するペナルティが強くなり、現在の方策は教師あり
学習済みモデルの方策に近づくように学習されます。逆に、βの値を小さくすると、事前学習
済みの方策からの乖離に対するペナルティが弱くなり、現在の方策は報酬モデルのスコアを最
大化することを優先するようになります。
b
の値は、タスクや目的に応じて適切に設定する必要があります。
b
の値を適切に設定する
ことで、人間の好みに適合しつつ、言語モデルとしての性能を維持した方策を学習できると考
えられています。
PPO損失関数
RLHFでは、報酬モデルから得られた報酬と、現在の方策と教師あり学習済みモデルの方策
とのKL ダイバージェンスを用いて、次のような目的関数
J
(
z
)を定義しました。
J
(
z
)=
E
(x, y)aD
r
[r
i
(
x
,
y
)-
bD
KL
(r
z
(y | x )||
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.