book

Deep Reinforcement Learning in Action

Name: Deep Reinforcement Learning in Action
ISBN: 9781617295430

by Alexander Zai, Brandon Brown

March 2020

Intermediate to advanced

384 pages

13h 10m

English

Manning Publications

Audiobook available

Read now

Unlock full access

Copyright
Brief Table of Contents
Table of Contents
Preface
Acknowledgments
About This Book
About the Authors
About the Cover Illustration
Part 1. Foundations
Chapter 1. What is reinforcement learning?
1.1. The “deep” in deep reinforcement learning1.2. Reinforcement learning1.3. Dynamic programming versus Monte Carlo1.4. The reinforcement learning framework1.5. What can I do with reinforcement learning?1.6. Why deep reinforcement learning?1.7. Our didactic tool: String diagrams1.8. What’s next?Summary

Chapter 2. Modeling reinforcement learning problems: Markov decision processes
2.1. String diagrams and our teaching methods2.2. Solving the multi-arm bandit2.3. Applying bandits to optimize ad placements2.4. Building networks with PyTorch2.5. Solving contextual bandits2.6. The Markov property2.7. Predicting future rewards: Value and policy functionsSummary
Chapter 3. Predicting the best states and actions: Deep Q-networks
3.1. The Q function3.2. Navigating with Q-learning3.3. Preventing catastrophic forgetting: Experience replay3.4. Improving stability with a target network3.5. ReviewSummary
Chapter 4. Learning to pick the best policy: Policy gradient methods
4.1. Policy function using neural networks4.2. Reinforcing good actions: The policy gradient algorithm4.3. Working with OpenAI Gym4.4. The REINFORCE algorithmSummary
Chapter 5. Tackling more complex problems with actor-critic methods
5.1. Combining the value and policy function5.2. Distributed training5.3. Advantage actor-critic5.4. N-step actor-criticSummary
Part 2. Above and beyond
Chapter 6. Alternative optimization methods: Evolutionary algorithms
6.1. A different approach to reinforcement learning6.2. Reinforcement learning with evolution strategies6.3. A genetic algorithm for CartPole6.4. Pros and cons of evolutionary algorithms6.5. Evolutionary algorithms as a scalable alternativeSummary
Chapter 7. Distributional DQN: Getting the full story
7.1. What’s wrong with Q-learning?7.2. Probability and statistics revisited7.3. The Bellman equation7.4. Distributional Q-learning7.5. Comparing probability distributions7.6. Dist-DQN on simulated data7.7. Using distributional Q-learning to play FreewaySummary
Chapter 8. Curiosity-driven exploration
8.1. Tackling sparse rewards with predictive coding8.2. Inverse dynamics prediction8.3. Setting up Super Mario Bros.8.4. Preprocessing and the Q-network8.5. Setting up the Q-network and policy function8.6. Intrinsic curiosity module8.7. Alternative intrinsic reward mechanismsSummary
Chapter 9. Multi-agent reinforcement learning
9.1. From one to many agents9.2. Neighborhood Q-learning9.3. The 1D Ising model9.4. Mean field Q-learning and the 2D Ising model9.5. Mixed cooperative-competitive gamesSummary
Chapter 10. Interpretable reinforcement learning: Attention and relational models
10.1. Machine learning interpretability with attention and relational biases10.2. Relational reasoning with attention10.3. Implementing self-attention for MNIST10.4. Multi-head attention and relational DQN10.5. Double Q-learning10.6. Training and attention visualizationSummary
Chapter 11. In conclusion: A review and roadmap
11.1. What did we learn?11.2. The uncharted topics in deep reinforcement learning11.3. The end
Appendix. Mathematics, deep learning, PyTorch
A.1. Linear algebra
A.2. Calculus
A.3. Deep learning
A.4. PyTorch
Reference list
Index
List of Figures
List of Tables
List of Listings

Content preview from Deep Reinforcement Learning in Action

List of Tables

Chapter 2. Modeling reinforcement learning problems: Markov decision processes

Table 2.1. Example of the side-by-side mathematics and pseudocode we use in this book

Table 2.2. The expected reward calculation in math and pseudocode

Table 2.3. Computing the best action, given the expected rewards

Table 2.4. The softmax equation

Table 2.5. The policy function

Table 2.6. The optimal policy

Table 2.7. The state-value function

Table 2.8. The action-value (Q) function

Chapter 3. Predicting the best states and actions: Deep Q-networks

Table 3.1. Q-learning update rule

Chapter 7. Distributional DQN: Getting the full story

Table 7.1. Frequentist versus Bayesian probabilities

Table 7.2. Computing an expected value from a probability ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781617295430Publisher Support Publisher Website Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Deep Reinforcement Learning in Action

by Alexander Zai, Brandon Brown

List of Tables

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.