February 2025
Intermediate
376 pages
9h 25m
Japanese
Content preview from 仕組みからわかる大規模言語モデル 生成AI時代のソフトウェア開発入門
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
学習における問題と対策
2.7
ニューラルネットワークの学習では、様々な問題が発生することがあります。LLM でも同
様であり、学習では次のような問題を避けるための対策が必要です。
過学習（Overfitting）
勾配消失（Vanishing Gradient）
勾配爆発（Exploding Gradient）
本節ではこれらの問題について説明し、対策について簡単に紹介します。
2.7.1
過学習（Overfitting）
過学習[Srivastava et al. 2014]とは、モデルが訓練データに過剰に適合し、未知のデータに
対する汎化性能が低下する現象です。過学習が起きるとモデルは訓練データには高い精度を示
しますが、テストデータや実際の応用場面では性能が低下してしまいます。
例えば、Zhangらによる実験 [H. Zhang et al. 2024] では、LLM の評価によく使われるデー
タセットであるGSM8K に対して過学習が起きていることを示唆しています。過学習が起きて
いるモデルでは、GSM8K と同じ難易度で新たに作成したGSM1K というデータセットに対し
て、GSM1Kに対して最大で 13% の性能低下が報告されています。
一般に過学習を防ぐための対策としては、次のような手法が用いられます。
正則化（Regularization）[Ng 2004]：モデルの複雑さにペナルティを与えることで、過
剰な適合を防ぐ手法です。L1正則化や、L2 正則化がよく用いられます。
ドロップアウト
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.