book

Deep Reinforcement Learning with Python - Second Edition

by Sudharsan Ravichandiran

September 2020

Intermediate to advanced

760 pages

18h 26m

English

Packt Publishing

Read now

Unlock full access

Who this book is forWhat this book coversTo get the most out of this bookGet in touch
Key elements of RLAgentEnvironmentState and actionRewardThe basic idea of RLThe RL algorithmRL agent in the grid worldHow RL differs from other ML paradigmsMarkov Decision ProcessesThe Markov property and Markov chainThe Markov Reward ProcessThe Markov Decision ProcessFundamental concepts of RLMath essentialsExpectationAction spacePolicyDeterministic policyStochastic policyEpisodeEpisodic and continuous tasksHorizonReturn and discount factorSmall discount factorLarge discount factorWhat happens when we set the discount factor to 0?What happens when we set the discount factor to 1?The value functionQ functionModel-based and model-free learningDifferent types of environmentsDeterministic and stochastic environmentsDiscrete and continuous environmentsEpisodic and non-episodic environmentsSingle and multi-agent environmentsApplications of RLRL glossarySummaryQuestionsFurther reading
Setting up our machineInstalling AnacondaInstalling the Gym toolkitCommon error fixesCreating our first Gym environmentExploring the environmentStatesActionsTransition probability and reward functionGenerating an episode in the Gym environmentAction selectionGenerating an episodeMore Gym environmentsClassic control environmentsState spaceAction spaceCart-Pole balancing with random policyAtari game environmentsGeneral environmentDeterministic environmentNo frame skippingState and action spaceAn agent playing the Tennis gameRecording the gameOther environmentsBox2DMuJoCoRoboticsToy textAlgorithmsEnvironment synopsisSummaryQuestionsFurther reading
The Bellman equationThe Bellman equation of the value functionThe Bellman equation of the Q functionThe Bellman optimality equationThe relationship between the value and Q functionsDynamic programmingValue iterationThe value iteration algorithmSolving the Frozen Lake problem with value iterationPolicy iterationAlgorithm – policy iterationSolving the Frozen Lake problem with policy iterationIs DP applicable to all environments? SummaryQuestions
Understanding the Monte Carlo methodPrediction and control tasksPrediction task Control taskMonte Carlo predictionMC prediction algorithmTypes of MC predictionFirst-visit Monte CarloEvery-visit Monte CarloImplementing the Monte Carlo prediction methodUnderstanding the blackjack gameThe blackjack environment in the Gym libraryEvery-visit MC prediction with the blackjack gameFirst-visit MC prediction with the blackjack gameIncremental mean updatesMC prediction (Q function)Monte Carlo controlMC control algorithmOn-policy Monte Carlo controlMonte Carlo exploring startsMonte Carlo with the epsilon-greedy policyImplementing on-policy MC controlOff-policy Monte Carlo controlIs the MC method applicable to all tasks?SummaryQuestions
TD learningTD predictionTD prediction algorithmPredicting the value of states in the Frozen Lake environmentTD controlOn-policy TD control – SARSAComputing the optimal policy using SARSAOff-policy TD control – Q learningComputing the optimal policy using Q learning The difference between Q learning and SARSAComparing the DP, MC, and TD methodsSummaryQuestionsFurther reading
The MAB problemCreating a bandit in the GymExploration strategiesEpsilon-greedySoftmax explorationUpper confidence bound Thompson samplingApplications of MABFinding the best advertisement banner using banditsCreating a dataset Initialize the variables Define the epsilon-greedy methodRun the bandit testContextual banditsSummaryQuestionsFurther reading
Biological and artificial neuronsANN and its layersInput layerHidden layerOutput layerExploring activation functionsThe sigmoid functionThe tanh functionThe Rectified Linear Unit functionThe softmax functionForward propagation in ANNsHow does an ANN learn?Putting it all togetherBuilding a neural network from scratchRecurrent Neural NetworksThe difference between feedforward networks and RNNsForward propagation in RNNsBackpropagating through timeLSTM to the rescueUnderstanding the LSTM cellWhat are CNNs?Convolutional layersStridesPaddingPooling layersFully connected layersThe architecture of CNNsGenerative adversarial networksBreaking down the generatorBreaking down the discriminatorHow do they learn, though?Architecture of a GANDemystifying the loss functionDiscriminator lossGenerator lossTotal lossSummaryQuestionsFurther reading
What is TensorFlow?Understanding computational graphs and sessionsSessionsVariables, constants, and placeholdersVariablesConstantsPlaceholders and feed dictionariesIntroducing TensorBoardCreating a name scopeHandwritten digit classification using TensorFlowImporting the required librariesLoading the datasetDefining the number of neurons in each layerDefining placeholdersForward propagationComputing loss and backpropagationComputing accuracyCreating a summaryTraining the modelVisualizing graphs in TensorBoardIntroducing eager executionMath operations in TensorFlowTensorFlow 2.0 and KerasBonjour KerasDefining the modelCompiling the modelTraining the modelEvaluating the modelMNIST digit classification using TensorFlow 2.0SummaryQuestionsFurther reading
What is DQN?Understanding DQNReplay bufferLoss functionTarget networkPutting it all togetherThe DQN algorithmPlaying Atari games using DQNArchitecture of the DQNGetting hands-on with the DQNPreprocess the game screenDefining the DQN classTraining the DQNThe double DQNThe double DQN algorithmDQN with prioritized experience replayTypes of prioritizationProportional prioritizationRank-based prioritizationCorrecting the biasThe dueling DQNUnderstanding the dueling DQNThe architecture of a dueling DQNThe deep recurrent Q networkThe architecture of a DRQNSummaryQuestionsFurther reading

Why policy-based methods?Policy gradient intuitionUnderstanding the policy gradientDeriving the policy gradientAlgorithm – policy gradientVariance reduction methodsPolicy gradient with reward-to-goAlgorithm – Reward-to-go policy gradientCart pole balancing with policy gradientComputing discounted and normalized rewardBuilding the policy networkTraining the networkPolicy gradient with baselineAlgorithm – REINFORCE with baselineSummaryQuestionsFurther reading
Overview of the actor-critic methodUnderstanding the actor-critic methodThe actor-critic algorithmAdvantage actor-critic (A2C)Asynchronous advantage actor-critic (A3C)The three AsThe architecture of A3CMountain car climbing using A3CCreating the mountain car environmentDefining the variablesDefining the actor-critic classDefining the worker classTraining the networkVisualizing the computational graphA2C revisitedSummaryQuestionsFurther reading
Deep deterministic policy gradient An overview of DDPGActor Critic DDPG componentsCritic networkActor networkPutting it all togetherAlgorithm – DDPGSwinging up a pendulum using DDPGCreating the Gym environmentDefining the variablesDefining the DDPG classTraining the network Twin delayed DDPG Key features of TD3Clipped double Q learningDelayed policy updatesTarget policy smoothingPutting it all togetherAlgorithm – TD3Soft actor-criticUnderstanding soft actor-criticV and Q functions with the entropy termComponents of SACCritic networkActor networkPutting it all together Algorithm – SACSummaryQuestionsFurther reading
Trust region policy optimizationMath essentialsThe Taylor seriesThe trust region methodThe conjugate gradient methodLagrange multipliers Importance sampling Designing the TRPO objective functionParameterizing the policiesSample-based estimationSolving the TRPO objective functionComputing the search directionPerforming a line search in the search directionAlgorithm – TRPOProximal policy optimizationPPO with a clipped objective Algorithm – PPO-clippedImplementing the PPO-clipped methodCreating the Gym environmentDefining the PPO classTraining the networkPPO with a penalized objectiveAlgorithm – PPO-penaltyActor-critic using Kronecker-factored trust region Math essentialsBlock matrixBlock diagonal matrixThe Kronecker productThe vec operatorProperties of the Kronecker productKronecker-Factored Approximate Curvature (K-FAC) K-FAC in actor-criticIncorporating the trust regionSummaryQuestionsFurther reading
Why distributional reinforcement learning?Categorical DQNPredicting the value distributionSelecting an action based on the value distributionTraining the categorical DQNProjection stepPutting it all togetherAlgorithm – categorical DQNPlaying Atari games using a categorical DQNDefining the variablesDefining the replay bufferDefining the categorical DQN classQuantile Regression DQNMath essentials Quantile Inverse CDF (quantile function)Understanding QR-DQNAction selectionLoss functionDistributed Distributional DDPGCritic networkActor networkAlgorithm – D4PGSummaryQuestionsFurther reading
Supervised imitation learning DAggerUnderstanding DAggerAlgorithm – DAggerDeep Q learning from demonstrationsPhases of DQfDPre-training phaseTraining phaseLoss function of DQfDAlgorithm – DQfDInverse reinforcement learningMaximum entropy IRLKey termsBack to maximum entropy IRLComputing the gradientAlgorithm – maximum entropy IRLGenerative adversarial imitation learningFormulation of GAIL SummaryQuestionsFurther reading
Installing Stable BaselinesCreating our first agent with Stable BaselinesEvaluating the trained agentStoring and loading the trained agentViewing the trained agent Putting it all togetherVectorized environments SubprocVecEnvDummyVecEnvIntegrating custom environmentsPlaying Atari games with a DQN and its variants Implementing DQN variantsLunar lander using A2CCreating a custom network Swinging up a pendulum using DDPGViewing the computational graph in TensorBoardTraining an agent to walk using TRPO Installing the MuJoCo environmentImplementing TRPORecording the videoTraining a cheetah bot to run using PPOMaking a GIF of a trained agentImplementing GAILSummaryQuestionsFurther reading
Meta reinforcement learningModel-agnostic meta learningUnderstanding MAMLMAML in a supervised learning settingMAML in a reinforcement learning settingHierarchical reinforcement learningMAXQ value function DecompositionImagination augmented agentsSummaryQuestionsFurther reading
Reinforcement learning algorithmValue IterationPolicy IterationFirst-Visit MC PredictionEvery-Visit MC PredictionMC Prediction – the Q FunctionMC Control MethodOn-Policy MC Control – Exploring startsOn-Policy MC Control – Epsilon-GreedyOff-Policy MC ControlTD PredictionOn-Policy TD Control – SARSAOff-Policy TD Control – Q LearningDeep Q LearningDouble DQNREINFORCE Policy GradientPolicy Gradient with Reward-To-GoREINFORCE with BaselineAdvantage Actor CriticAsynchronous Advantage Actor-CriticDeep Deterministic Policy GradientTwin Delayed DDPGSoft Actor-Critic Trust Region Policy OptimizationPPO-ClippedPPO-PenaltyCategorical DQNDistributed Distributional DDPG DAggerDeep Q learning from demonstrationsMaxEnt Inverse Reinforcement LearningMAML in Reinforcement Learning
Chapter 1 – Fundamentals of Reinforcement LearningChapter 2 – A Guide to the Gym ToolkitChapter 3 – The Bellman Equation and Dynamic ProgrammingChapter 4 – Monte Carlo MethodsChapter 5 – Understanding Temporal Difference LearningChapter 6 – Case Study – The MAB ProblemChapter 7 – Deep Learning FoundationsChapter 8 – A Primer on TensorFlowChapter 9 – Deep Q Network and Its VariantsChapter 10 – Policy Gradient MethodChapter 11 – Actor-Critic Methods – A2C and A3CChapter 12 – Learning DDPG, TD3, and SACChapter 13 – TRPO, PPO, and ACKTR MethodsChapter 14 – Distributional Reinforcement LearningChapter 15 – Imitation Learning and Inverse RLChapter 16 – Deep Reinforcement Learning with Stable BaselinesChapter 17 – Reinforcement Learning Frontiers

Content preview from Deep Reinforcement Learning with Python - Second Edition

13 TRPO, PPO, and ACKTR Methods

In this chapter, we will learn two interesting state-of-art policy gradient algorithms: trust region policy optimization and proximal policy optimization. Both of these algorithms act as an improvement to the policy gradient algorithm (REINFORCE with baseline) we learned in Chapter 10, Policy Gradient Method.

We begin the chapter by understanding the Trust Region Policy Optimization (TRPO) method and how it acts as an improvement to the policy gradient method. Later we will understand several essential math concepts that are required to understand TRPO. Following this, we will learn how to design and solve the TRPO objective function. At the end of the section, we will understand how the TRPO algorithm works step ...