April 2020
Intermediate to advanced
816 pages
18h 32m
Chinese
Content preview from 精通機器學習
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
680
|
第十九章：大規模訓練與部署 TensorFlow 模型
非同步更新
使用非同步更新時
，
當複本算好梯度時
，
它會立刻用那些梯度來更新模型參
數
。
這種做法沒有整合的動作
（
它免於使用圖
19-19
的
「
平均
」
步驟
），
也沒有同步的動
作
。
複本彼此是獨立的
。
因為不需要等待其他的複本
，
這種做法每分鐘可執行更多訓練步
驟
。
此外
，
雖然這種做法在每一步時
，
仍然要將參數複製到每一個裝置
，
但這會發生在各
個複本的不同時刻
，
所以可以降低頻寬飽和的風險
。
資料平行化和非同步更新是很誘人的選項
，
因為它們很簡單
、
沒有同步延遲
，
而且更妥
善地運用頻寬
。
雖然它在實務上有很好的效果
，
但它能夠奏效幾乎是個奇蹟
！
事實上
，
當
一個複本用一些參數值來完成梯度計算之後
，
那些參數會被其他複本更新很多次
（
如果
有
N
個複本
，
平均
N – 1
次
），
而且無法保證算出來的梯度仍然指往正確的方向
（
見圖
19-
20）。
當梯度嚴重過期時
，
它們稱為
舊梯度
（stale gradient）：
它們可能會減緩收斂速度
、
引入雜訊與搖擺效應
（
學習曲線可能有暫時的振盪
），
它們甚至會讓訓練演算法發散
。
被其他複本更新
梯度是在
這裡計算的...
...但它們
被用在這裡
代價
∇
θ
∇
θ
哎呀，
上坡！
舊梯度
圖
19-20
在使用非同步更新時產生的陳舊梯度
你可以用幾種方式來降低陳舊梯度的影響
：
•
降低學習速度
。
•
移除舊梯度或縮小它們
。
•
調整小批次大小
。
•
在前幾期只使用一個複本
（
稱為
暖機階段
）。
陳舊梯度在訓練剛開始往往較具殺傷
力
，
因為此時的梯度通常很大
，
而且參數尚未在代價函數的山谷中穩定下來 ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.