July 2018
Intermediate to advanced
310 pages
8h 21m
Chinese
Content preview from 深度学习入门 : 基于Python的理论与实现
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
第 6章 与学习相关的技巧
170
图6-5 基于Momentum 的最优化的更新路径
10
10
50
−
5
−
10
5
0
−5
−10
y
x
Momentum
6.1.5
AdaGrad
在神经网络的学习中，学习率（数学式中记为
η
）的值很重要。学习率过小，
会导致学习花费过多时间；反过来，学习率过大，则会导致学习发散而不能
正确进行。
在关于学习率的有效技巧中，有一种被称为学习率衰减（
learning rate
decay
）的方法，即随着学习的进行，使学习率逐渐减小。实际上，一开始“多”
学，然后逐渐“少”学的方法，在神经网络的学习中经常被使用。
逐渐减小学习率的想法，相当于将“全体”参数的学习率值一起降低。
而
AdaGrad
[6]
进一步发展了这个想法，针对“一个一个”的参数，赋予其“定
制”的值。
AdaGrad
会为参数的每个元素适当地调整学习率，与此同时进行学习
（
AdaGrad
的
Ada
来自英文单词
Adaptive
，即“适当的”的意思）。下面，让
我们用数学式表示
AdaGrad
的更新方法。
6.1 参数的更新
171
（6.5）
（6.6）
和前面的
SGD
一样，
W
表示要更新的权重参数，
表示损失函数关
于
W
的梯度，
η
表示学习率。这里新出现了变量
h
，如式
(
6.5
)
所示，它保
存了以前的所有梯度值的平方和（式（6.5）中
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.