February 2025
Intermediate
376 pages
9h 25m
Japanese
Content preview from 仕組みからわかる大規模言語モデル 生成AI時代のソフトウェア開発入門
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
2
2.7.2
勾配消失（Vanishing Gradient）
勾配消失[Hochreiter 1998] とは、誤差逆伝播法において勾配が層を経るごとに指数関数的
に小さくなる現象です。勾配消失が起きると深い層のパラメータが更新されにくくなり、学習
が進まなくなります。LLMでは、多数の層を持つため、勾配消失への対策が重要となります。
勾配消失を防ぐために、次のような手法が用いられます。
ReLU活性化関数 [Nair et al. 2010]：勾配消失を起こしにくい活性化関数です。
残差接続（Residual Connection） [He et al. 2015a]：層の入力を出力に直接加算する接
続です。勾配が直接伝播されるため、勾配消失を防ぐことができます。
Layer Normalization [Ba et al. 2016]：各層の入力を正規化する手法です。勾配消失を
防ぐことができます。
これらの手法は1.3 で紹介したTransformer にも適用されており、勾配消失を防ぐために効
果的とされています。
2.7.3
勾配爆発（Exploding Gradient）
勾配爆発[Pascanu et al. 2013]とは、誤差逆伝播法において、勾配が層を経るごとに指数関
数的に大きくなる現象です。勾配爆発が起きるとパラメータが不安定になり、学習が進まなく
なります。LLMでは、多数の層を持つため、勾配爆発への対策が重要です。
勾配爆発を防ぐための対策としては、次のような手法が用いられます。 ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.