book

Grokking Deep Reinforcement Learning

Name: Grokking Deep Reinforcement Learning
Author: Miguel Morales
ISBN: 9781617295454

by Miguel Morales

December 2020

Intermediate to advanced

472 pages

13h 36m

English

Manning Publications

Read now

Unlock full access

Grokking Deep Reinforcement Learning
Copyright
dedication
contents
front matter
forewordprefaceacknowledgmentsabout this bookWho should read this bookHow this book is organized: a roadmapAbout the codeliveBook discussion forumabout the author
1 Introduction to deep reinforcement learning
What is deep reinforcement learning?Deep reinforcement learning is a machine learning approach to artificial intelligenceDeep reinforcement learning is concerned with creating computer programsDeep reinforcement learning agents can solve problems that require intelligenceDeep reinforcement learning agents improve their behavior through trial-and-error learningDeep reinforcement learning agents learn from sequential feedbackDeep reinforcement learning agents learn from evaluative feedbackDeep reinforcement learning agents learn from sampled feedbackDeep reinforcement learning agents use powerful non-linear function approximationThe past, present, and future of deep reinforcement learningRecent history of artificial intelligence and deep reinforcement learningArtificial intelligence wintersThe current state of artificial intelligenceProgress in deep reinforcement learningOpportunities aheadThe suitability of deep reinforcement learningWhat are the pros and cons?Deep reinforcement learning’s strengthsDeep reinforcement learning’s weaknessesSetting clear two-way expectationsWhat to expect from the book?How to get the most out of this bookDeep reinforcement learning development environmentSummary
2 Mathematical foundations of reinforcement learning
Components of reinforcement learningExamples of problems, agents, and environmentsThe agent: The decision makerThe environment: Everything elseAgent-environment interaction cycleMDPs: The engine of the environmentStates: Specific configurations of the environmentActions: A mechanism to influence the environmentTransition function: Consequences of agent actionsReward signal: Carrots and sticksHorizon: Time changes what’s optimalDiscount: The future is uncertain, value it lessExtensions to MDPsPutting it all togetherSummary
3 Balancing immediate and long-term goals
The objective of a decision-making agentPolicies: Per-state action prescriptionsState-value function: What to expect from here?Action-value function: What should I expect from here if I do this?Action-advantage function: How much better if I do that?OptimalityPlanning optimal sequences of actionsPolicy evaluation: Rating policiesPolicy improvement: Using ratings to get betterPolicy iteration: Improving upon improved behaviorsValue iteration: Improving behaviors earlySummary
4 Balancing the gathering and use of information
The challenge of interpreting evaluative feedbackBandits: Single-state decision problemsRegret: The cost of explorationApproaches to solving MAB environmentsGreedy: Always exploitRandom: Always exploreEpsilon-greedy: Almost always greedy and sometimes randomDecaying epsilon-greedy: First maximize exploration, then exploitationOptimistic initialization: Start off believing it’s a wonderful worldStrategic explorationSoftmax: Select actions randomly in proportion to their estimatesUCB: It’s not about optimism, it’s about realistic optimismThompson sampling: Balancing reward and riskSummary
5 Evaluating agents’ behaviors
Learning to estimate the value of policiesFirst-visit Monte Carlo: Improving estimates after each episodeEvery-visit Monte Carlo: A different way of handling state visitsTemporal-difference learning: Improving estimates after each stepLearning to estimate from multiple stepsN-step TD learning: Improving estimates after a couple of stepsForward-view TD(λ): Improving estimates of all visited statesTD(λ): Improving estimates of all visited states after each stepSummary

6 Improving agents’ behaviors
The anatomy of reinforcement learning agentsMost agents gather experience samplesMost agents estimate somethingMost agents improve a policyGeneralized policy iterationLearning to improve policies of behaviorMonte Carlo control: Improving policies after each episodeSARSA: Improving policies after each stepDecoupling behavior from learningQ-learning: Learning to act optimally, even if we choose not toDouble Q-learning: A max of estimates for an estimate of a maxSummary
7 Achieving goals more effectively and efficiently
Learning to improve policies using robust targetsSARSA(λ): Improving policies after each step based on multi-step estimatesWatkins’s Q(λ): Decoupling behavior from learning, againAgents that interact, learn, and planDyna-Q: Learning sample modelsTrajectory sampling: Making plans for the immediate futureSummary
8 Introduction to value-based deep reinforcement learning
The kind of feedback deep reinforcement learning agents useDeep reinforcement learning agents deal with sequential feedbackBut, if it isn’t sequential, what is it?Deep reinforcement learning agents deal with evaluative feedbackBut, if it isn’t evaluative, what is it?Deep reinforcement learning agents deal with sampled feedbackBut, if it isn’t sampled, what is it?Introduction to function approximation for reinforcement learningReinforcement learning problems can have high-dimensional state and action spacesReinforcement learning problems can have continuous state and action spacesThere are advantages when using function approximationNFQ: The first attempt at value-based deep reinforcement learningFirst decision point: Selecting a value function to approximateSecond decision point: Selecting a neural network architectureThird decision point: Selecting what to optimizeFourth decision point: Selecting the targets for policy evaluationFifth decision point: Selecting an exploration strategySixth decision point: Selecting a loss functionSeventh decision point: Selecting an optimization methodThings that could (and do) go wrongSummary
9 More stable value-based methods
DQN: Making reinforcement learning more like supervised learningCommon problems in value-based deep reinforcement learningUsing target networksUsing larger networksUsing experience replayUsing other exploration strategiesDouble DQN: Mitigating the overestimation of action-value functionsThe problem of overestimation, take twoSeparating action selection from action evaluationA solutionA more practical solutionA more forgiving loss functionThings we can still improve onSummary
10 Sample-efficient value-based methods
Dueling DDQN: A reinforcement-learning-aware neural network architectureReinforcement learning isn’t a supervised learning problemNuances of value-based deep reinforcement learning methodsAdvantage of using advantagesA reinforcement-learning-aware architectureBuilding a dueling networkReconstructing the action-value functionContinuously updating the target networkWhat does the dueling network bring to the table?PER: Prioritizing the replay of meaningful experiencesA smarter way to replay experiencesThen, what’s a good measure of “important” experiences?Greedy prioritization by TD errorSampling prioritized experiences stochasticallyProportional prioritizationRank-based prioritizationPrioritization biasSummary
11 Policy-gradient and actor-critic methods
REINFORCE: Outcome-based policy learningIntroduction to policy-gradient methodsAdvantages of policy-gradient methodsLearning policies directlyReducing the variance of the policy gradientVPG: Learning a value functionFurther reducing the variance of the policy gradientLearning a value functionEncouraging explorationA3C: Parallel policy updatesUsing actor-workersUsing n-step estimatesNon-blocking model updatesGAE: Robust advantage estimationGeneralized advantage estimationA2C: Synchronous policy updatesWeight-sharing modelRestoring order in policy updatesSummary
12 Advanced actor-critic methods
DDPG: Approximating a deterministic policyDDPG uses many tricks from DQNLearning a deterministic policyExploration with deterministic policiesTD3: State-of-the-art improvements over DDPGDouble learning in DDPGSmoothing the targets used for policy updatesDelaying updatesSAC: Maximizing the expected return and entropyAdding the entropy to the Bellman equationsLearning the action-value functionLearning the policyAutomatically tuning the entropy coefficientPPO: Restricting optimization stepsUsing the same actor-critic architecture as A2CBatching experiencesClipping the policy updatesClipping the value function updatesSummary
13 Toward artificial general intelligence
What was covered and what notably wasn’t?Markov decision processesPlanning methodsBandit methodsTabular reinforcement learningValue-based deep reinforcement learningPolicy-based and actor-critic deep reinforcement learningAdvanced actor-critic techniquesModel-based deep reinforcement learningDerivative-free optimization methodsMore advanced concepts toward AGIWhat is AGI, again?Advanced exploration strategiesInverse reinforcement learningTransfer learningMulti-task learningCurriculum learningMeta learningHierarchical reinforcement learningMulti-agent reinforcement learningExplainable AI, safety, fairness, and ethical standardsWhat happens next?How to use DRL to solve custom problemsGoing forwardGet yourself out there! Now!Summary
index

Content preview from Grokking Deep Reinforcement Learning

9 More stable value-based methods

In this chapter

You will improve on the methods you learned in the previous chapter by making them more stable and therefore less prone to divergence.
You will explore advanced value-based deep reinforcement learning methods, and the many components that make value-based methods better.
You will solve the cart-pole environment in a fewer number of samples, and with more reliable and consistent results.

Let thy step be slow and steady, that thou stumble not.

— Tokugawa Ieyasu Founder and first shōgun of the Tokugawa shogunate of Japan and one of the three unifiers of Japan

In the last chapter, you learned about value-based deep reinforcement learning. NFQ, the algorithm we developed, is a simple solution to ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Deep Reinforcement Learning Hands-On - Third Edition

Publisher Resources

ISBN: 9781617295454Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Grokking Deep Reinforcement Learning

by Miguel Morales

9 More stable value-based methods

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.