Skip to Main Content
Deep Reinforcement Learning Hands-On
book

Deep Reinforcement Learning Hands-On

by Oleg Vasilev, Maxim Lapan, Martijn van Otterlo, Mikhail Yurushkin, Basem O. F. Alijla
June 2018
Intermediate to advanced content levelIntermediate to advanced
546 pages
13h 30m
English
Packt Publishing
Content preview from Deep Reinforcement Learning Hands-On

Proximal Policy Optimization

Historically, this method came from the OpenAI team and was proposed long after TRPO (which is from 2015), but PPO is much simpler than TRPO, so we'll start from it. The paper in which it was proposed is by John Schulman et al and called Proximal Policy Optimization Algorithms, published in 2017 (arXiv:1707.06347).

The core improvement over the classical Asynchronous Advantage Actor-Critic (A3C) method is to change the expression used to estimate the PG. Instead of the gradient of logarithm probability of the action taken, the PPO method uses a different objective: the ratio between the new and the old policy scaled by the advantages.

In math form, the old A3C objective could be written as . The new objective proposed ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

Grokking Deep Reinforcement Learning

Grokking Deep Reinforcement Learning

Miguel Morales

Publisher Resources

ISBN: 9781788834247Supplemental Content