February 2025
Intermediate
376 pages
9h 25m
Japanese
Content preview from 仕組みからわかる大規模言語モデル 生成AI時代のソフトウェア開発入門
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
2
図2.4.1
図2.4.1 に、RLHF におけるエージェントと環境の相互作用を示します。選択された行動に
対して環境は報酬を与え、エージェントはその報酬を最大化するように学習します。RLHFで
は、報酬モデルを用いて人間のフィードバックを数値化したスコアを報酬として使用します。
このため、強化学習を行う事前準備として、報酬モデルの構築とその事前学習が必要となりま
す。
以降、本節では報酬モデルについて説明した後、RLHFの学習プロセスについて詳しく説明
します。
2.4.3
報酬モデル
強化学習では、報酬を最大化することが目的となります。LLMの場合、人間の好みにより
適合した出力が得られるほど報酬が高くなります。しかし、人間の好みに適合しているか否か
を毎回人手でチェックすることは現実的ではありません。そこで、自動的に報酬を計算するた
めの報酬モデルを事前に準備し、使用します。
報酬モデルは、プロンプトとLLM の生成したテキストを入力とし、そのテキストの質や適
切さを評価するスコアを出力します。出力するスコアは高いほど人間の好みに適合しているこ
とを示し、低いほど適合していないことを示します。RLHFでは、強化学習の前に報酬モデル
を事前学習させ、そのスコアを用いてLLM の学習を行います。
報酬モデルは、通常、事前学習済みの言語モデルをベースとし、最後に回帰層を追加して訓 ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.