book

Deep Reinforcement Learning in Action

Name: Deep Reinforcement Learning in Action
ISBN: 9781617295430

by Alexander Zai, Brandon Brown

March 2020

Intermediate to advanced

384 pages

13h 10m

English

Manning Publications

Audiobook available

Read now

Unlock full access

Copyright
Brief Table of Contents
Table of Contents
Preface
Acknowledgments
About This Book
About the Authors
About the Cover Illustration
Part 1. Foundations
Chapter 1. What is reinforcement learning?
1.1. The “deep” in deep reinforcement learning1.2. Reinforcement learning1.3. Dynamic programming versus Monte Carlo1.4. The reinforcement learning framework1.5. What can I do with reinforcement learning?1.6. Why deep reinforcement learning?1.7. Our didactic tool: String diagrams1.8. What’s next?Summary

Chapter 2. Modeling reinforcement learning problems: Markov decision processes
2.1. String diagrams and our teaching methods2.2. Solving the multi-arm bandit2.3. Applying bandits to optimize ad placements2.4. Building networks with PyTorch2.5. Solving contextual bandits2.6. The Markov property2.7. Predicting future rewards: Value and policy functionsSummary
Chapter 3. Predicting the best states and actions: Deep Q-networks
3.1. The Q function3.2. Navigating with Q-learning3.3. Preventing catastrophic forgetting: Experience replay3.4. Improving stability with a target network3.5. ReviewSummary
Chapter 4. Learning to pick the best policy: Policy gradient methods
4.1. Policy function using neural networks4.2. Reinforcing good actions: The policy gradient algorithm4.3. Working with OpenAI Gym4.4. The REINFORCE algorithmSummary
Chapter 5. Tackling more complex problems with actor-critic methods
5.1. Combining the value and policy function5.2. Distributed training5.3. Advantage actor-critic5.4. N-step actor-criticSummary
Part 2. Above and beyond
Chapter 6. Alternative optimization methods: Evolutionary algorithms
6.1. A different approach to reinforcement learning6.2. Reinforcement learning with evolution strategies6.3. A genetic algorithm for CartPole6.4. Pros and cons of evolutionary algorithms6.5. Evolutionary algorithms as a scalable alternativeSummary
Chapter 7. Distributional DQN: Getting the full story
7.1. What’s wrong with Q-learning?7.2. Probability and statistics revisited7.3. The Bellman equation7.4. Distributional Q-learning7.5. Comparing probability distributions7.6. Dist-DQN on simulated data7.7. Using distributional Q-learning to play FreewaySummary
Chapter 8. Curiosity-driven exploration
8.1. Tackling sparse rewards with predictive coding8.2. Inverse dynamics prediction8.3. Setting up Super Mario Bros.8.4. Preprocessing and the Q-network8.5. Setting up the Q-network and policy function8.6. Intrinsic curiosity module8.7. Alternative intrinsic reward mechanismsSummary
Chapter 9. Multi-agent reinforcement learning
9.1. From one to many agents9.2. Neighborhood Q-learning9.3. The 1D Ising model9.4. Mean field Q-learning and the 2D Ising model9.5. Mixed cooperative-competitive gamesSummary
Chapter 10. Interpretable reinforcement learning: Attention and relational models
10.1. Machine learning interpretability with attention and relational biases10.2. Relational reasoning with attention10.3. Implementing self-attention for MNIST10.4. Multi-head attention and relational DQN10.5. Double Q-learning10.6. Training and attention visualizationSummary
Chapter 11. In conclusion: A review and roadmap
11.1. What did we learn?11.2. The uncharted topics in deep reinforcement learning11.3. The end
Appendix. Mathematics, deep learning, PyTorch
A.1. Linear algebra
A.2. Calculus
A.3. Deep learning
A.4. PyTorch
Reference list
Index
List of Figures
List of Tables
List of Listings

Content preview from Deep Reinforcement Learning in Action

Chapter 10. Interpretable reinforcement learning: Attention and relational models

This chapter covers

Implementing a relational reinforcement algorithm using the popular self-attention model
Visualizing attention maps to better interpret the reasoning of an RL agent
Reasoning about model invariance and equivariance
Incorporating double Q-learning to improve the stability of training

Hopefully by this point you have come to appreciate just how powerful the combination of deep learning and reinforcement learning is for solving tasks previously thought to be the exclusive domain of humans. Deep learning is a class of powerful learning algorithms that can comprehend and reason through complex patterns and data, and reinforcement learning is the ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781617295430Publisher Support Publisher Website Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Deep Reinforcement Learning in Action

by Alexander Zai, Brandon Brown

Chapter 10. Interpretable reinforcement learning: Attention and relational models

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.