book

Deep Reinforcement Learning Hands-On - Second Edition

Name: Deep Reinforcement Learning Hands-On - Second Edition
Author: Maxim Lapan
ISBN: 9781838826994

by Maxim Lapan

January 2020

Intermediate to advanced

826 pages

21h 1m

English

Packt Publishing

Read now

Unlock full access

Preface
Why I wrote this bookThe approachWho this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedGet in touchReviews
What Is Reinforcement Learning?
Supervised learningUnsupervised learningReinforcement learningRL's complicationsRL formalismsRewardThe agentThe environmentActionsObservationsThe theoretical foundations of RLMarkov decision processesThe Markov processMarkov reward processesAdding actionsPolicySummary
OpenAI Gym
The anatomy of the agentHardware and software requirementsThe OpenAI Gym APIThe action spaceThe observation spaceThe environmentCreating an environmentThe CartPole sessionThe random CartPole agentExtra Gym functionality – wrappers and monitorsWrappersMonitorSummary
Deep Learning with PyTorch
TensorsThe creation of tensorsScalar tensorsTensor operationsGPU tensorsGradientsTensors and gradientsNN building blocksCustom layersThe final glue – loss functions and optimizersLoss functionsOptimizersMonitoring with TensorBoardTensorBoard 101Plotting stuffExample – GAN on Atari imagesPyTorch IgniteIgnite conceptsSummary
The Cross-Entropy Method
The taxonomy of RL methodsThe cross-entropy method in practiceThe cross-entropy method on CartPoleThe cross-entropy method on FrozenLakeThe theoretical background of the cross-entropy methodSummary
Tabular Learning and the Bellman Equation
Value, state, and optimalityThe Bellman equation of optimalityThe value of the actionThe value iteration methodValue iteration in practiceQ-learning for FrozenLakeSummary
Deep Q-Networks
Real-life value iterationTabular Q-learningDeep Q-learningInteraction with the environmentSGD optimizationCorrelation between stepsThe Markov propertyThe final form of DQN trainingDQN on PongWrappersThe DQN modelTrainingRunning and performanceYour model in actionThings to trySummary
Higher-Level RL Libraries
Why RL libraries?The PTAN libraryAction selectorsThe agentDQNAgentPolicyAgentExperience sourceToy environmentThe ExperienceSource classExperienceSourceFirstLastExperience replay buffersThe TargetNet classIgnite helpersThe PTAN CartPole solverOther RL librariesSummary
DQN Extensions
Basic DQNCommon libraryImplementationResultsN-step DQNImplementationResultsDouble DQNImplementationResultsNoisy networksImplementationResultsPrioritized replay bufferImplementationResultsDueling DQNImplementationResultsCategorical DQNImplementationResultsCombining everythingResultsSummaryReferences
Ways to Speed up RL
Why speed mattersThe baselineThe computation graph in PyTorchSeveral environmentsPlay and train in separate processesTweaking wrappersBenchmark summaryGoing hardcore: CuLESummaryReferences

Stocks Trading Using RL
TradingDataProblem statements and key decisionsThe trading environmentModelsTraining codeResultsThe feed-forward modelThe convolution modelThings to trySummary
Policy Gradients – an Alternative
Values and policyWhy the policy?Policy representationPolicy gradientsThe REINFORCE methodThe CartPole exampleResultsPolicy-based versus value-based methodsREINFORCE issuesFull episodes are requiredHigh gradients varianceExplorationCorrelation between samplesPolicy gradient methods on CartPoleImplementationResultsPolicy gradient methods on PongImplementationResultsSummary
The Actor-Critic Method
Variance reductionCartPole varianceActor-criticA2C on PongA2C on Pong resultsTuning hyperparametersLearning rateEntropy betaCount of environmentsBatch sizeSummary
Asynchronous Advantage Actor-Critic
Correlation and sample efficiencyAdding an extra A to A2CMultiprocessing in PythonA3C with data parallelismImplementationResultsA3C with gradients parallelismImplementationResultsSummary
Training Chatbots with RL
An overview of chatbotsChatbot trainingThe deep NLP basicsRNNsWord embeddingThe Encoder-Decoder architectureSeq2seq trainingLog-likelihood trainingThe bilingual evaluation understudy (BLEU) scoreRL in seq2seqSelf-critical sequence trainingChatbot exampleThe example structureModules: cornell.py and data.pyBLEU score and utils.pyModelDataset explorationTraining: cross-entropyImplementationResultsTraining: SCSTImplementationResultsModels tested on dataTelegram botSummary
The TextWorld Environment
Interactive fictionThe environmentInstallationGame generationObservation and action spacesExtra game informationBaseline DQNObservation preprocessingEmbeddings and encodersThe DQN model and the agentTraining codeTraining resultsThe command generation modelImplementationPretraining resultsDQN training codeThe result of DQN trainingSummary
Web Navigation
Web navigationBrowser automation and RLThe MiniWoB benchmarkOpenAI UniverseInstallationActions and observationsEnvironment creationMiniWoB stabilityThe simple clicking approachGrid actionsExample overviewThe modelThe training codeStarting containersThe training processChecking the learned policyIssues with simple clickingHuman demonstrationsRecording the demonstrationsThe recording formatTraining using demonstrationsResultsThe tic-tac-toe problemAdding text descriptionsImplementationResultsThings to trySummary
Continuous Action Space
Why a continuous space?The action spaceEnvironmentsThe A2C methodImplementationResultsUsing models and recording videosDeterministic policy gradientsExplorationImplementationResultsRecording videosDistributional policy gradientsArchitectureImplementationResultsVideo recordingsThings to trySummary
RL in Robotics
Robots and roboticsRobot complexitiesThe hardware overviewThe platformThe sensorsThe actuatorsThe frameThe first training objectiveThe emulator and the modelThe model definition fileThe robot classDDPG training and resultsControlling the hardwareMicroPythonDealing with sensorsThe I2C busSensor initialization and readingSensor classes and timer readingObservationsDriving servosMoving the model to hardwareThe model exportBenchmarksCombining everythingPolicy experimentsSummary
Trust Regions – PPO, TRPO, ACKTR, and SAC
RoboschoolThe A2C baselineImplementationResultsVideo recordingPPOImplementationResultsTRPOImplementationResultsACKTRImplementationResultsSACImplementationResultsSummary
Black-Box Optimization in RL
Black-box methodsEvolution strategiesES on CartPoleResultsES on HalfCheetahImplementationResultsGenetic algorithmsGA on CartPoleResultsGA tweaksDeep GANovelty searchGA on HalfCheetahResultsSummaryReferences
Advanced Exploration
Why exploration is importantWhat's wrong with ε-greedy?Alternative ways of explorationNoisy networksCount-based methodsPrediction-based methodsMountainCar experimentsThe DQN method with ε-greedyThe DQN method with noisy networksThe DQN method with state countsThe proximal policy optimization methodThe PPO method with noisy networksThe PPO method with count-based explorationThe PPO method with network distillationAtari experimentsThe DQN method with ε-greedyThe classic PPO methodThe PPO method with network distillationThe PPO method with noisy networksSummaryReferences
Beyond Model-Free – Imagination
Model-based methodsModel-based versus model-freeModel imperfectionsThe imagination-augmented agentThe EMThe rollout policyThe rollout encoderThe paper's resultsI2A on Atari BreakoutThe baseline A2C agentEM trainingThe imagination agentThe I2A modelThe Rollout encoderThe training of I2AExperiment resultsThe baseline agentTraining EM weightsTraining with the I2A modelSummaryReferences
AlphaGo Zero
Board gamesThe AlphaGo Zero methodOverviewMCTSSelf-playTraining and evaluationThe Connect 4 botThe game modelImplementing MCTSThe modelTrainingTesting and comparisonConnect 4 resultsSummaryReferences
RL in Discrete Optimization
RL's reputationThe Rubik's Cube and combinatorial optimizationOptimality and God's numberApproaches to cube solvingData representationActionsStatesThe training processThe NN architectureThe trainingThe model applicationThe paper's resultsThe code outlineCube environmentsTrainingThe search processThe experiment resultsThe 2×2 cubeThe 3×3 cubeFurther improvements and experimentsSummary
Multi-agent RL
Multi-agent RL explainedForms of communicationThe RL approachThe MAgent environmentInstallationAn overviewA random environmentDeep Q-network for tigersTraining and resultsCollaboration by the tigersTraining both tigers and deerThe battle between equal actorsSummary
Other Books You May Enjoy
Index

Content preview from Deep Reinforcement Learning Hands-On - Second Edition

19 Trust Regions – PPO, TRPO, ACKTR, and SAC

Next, we will take a look at the approaches used to improve the stability of the stochastic policy gradient method. Some attempts have been made to make the policy improvement more stable, and in this chapter, we will focus on three methods:

Proximal policy optimization (PPO)
Trust region policy optimization (TRPO)
Advantage actor-critic (A2C) using Kronecker-factored trust region (ACKTR).

In addition, we will compare those methods to a relatively new off-policy method called soft actor-critic (SAC), which is a development of the deep deterministic policy gradients (DDPG) method described in Chapter 17, Continuous Action Space. To compare them to the A2C baseline, we will use several environments ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Deep Reinforcement Learning Hands-On - Third Edition

Publisher Resources

ISBN: 9781838826994

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Deep Reinforcement Learning Hands-On - Second Edition

by Maxim Lapan

19

Trust Regions – PPO, TRPO, ACKTR, and SAC

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.