book

Mastering Reinforcement Learning with Python

Name: Mastering Reinforcement Learning with Python
Author: Enes Bilgin
ISBN: 9781838644147

by Enes Bilgin

December 2020

Intermediate to advanced

544 pages

11h 55m

English

Packt Publishing

Read now

Unlock full access

Mastering Reinforcement Learning with Python
Why subscribe?ContributorsAbout the authorAbout the reviewersPackt is searching for authors like you
Preface
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedGet in touchReviews
Section 1: Reinforcement Learning Foundations
Chapter 1: Introduction to Reinforcement Learning
Why reinforcement learning?The three paradigms of MLSupervised learningUnsupervised learningReinforcement learningRL application areas and success stories GamesRobotics and autonomous systemsSupply chainManufacturingPersonalization and recommender systemsSmart citiesElements of a RL problemRL conceptsCasting Tic-Tac-Toe as a RL problemSetting up your RL environmentHardware requirementsOperating systemSoftware toolboxSummaryReferences
Chapter 2: Multi-Armed Bandits
Exploration-Exploitation Trade-OffWhat is a MAB?Problem definitionExperimenting with a simple MAB problemCase study: Online advertisingA/B/n testingNotationApplication to the online advertising scenarioAdvantages and Disadvantages of A/B/n Testingε-greedy actionsApplication to the online advertising scenarioAdvantages and disadvantages of ε-greedy actionsAction selection using upper confidence boundsApplication to the online advertising scenarioAdvantages and disadvantages of using UCBsThompson (Posterior) samplingApplication to the online advertising scenarioAdvantages and disadvantages of Thompson samplingSummaryReferences
Chapter 3: Contextual Bandits
Why we need function approximationsUsing function approximation for contextCase study: Contextual online advertising with synthetic user dataFunction approximation with regularized logistic regressionObjective: Regret minimizationSolving the online advertising problemUsing function approximation for actionCase study: Contextual online advertising with user data from U.S. CensusFunction approximation using a neural networkCalculating the regretSolving the online advertising problemOther applications of multi-armed and contextual banditsRecommender systemsWebpage / app feature designHealthcareDynamic pricingFinanceControl systems tuningSummaryReferences
Chapter 4: Makings of a Markov Decision Process
Starting with Markov chainsStochastic processes with Markov propertyClassification of states in a Markov chainExample: -step behavior in the grid worldExample: Sample path in an ergodic Markov chainSemi-Markov processes and continuous-time Markov chainsIntroducing the reward: Markov reward processAttaching rewards to the grid world exampleRelations between average rewards with different initializationsReturn, discount and state valuesAnalytically Calculating the State ValuesEstimating the state values iterativelyBringing the action in: Markov decision processDefinitionGrid world as a Markov decision processState-value functionAction-value functionOptimal state-value and action-value functionsBellman optimalityPartially observable Markov decision processSummaryExercisesReferences
Chapter 5: Solving the Reinforcement Learning Problem
Exploring dynamic programmingExample use case: Inventory replenishment of a food truckPolicy evaluationPolicy iterationValue iterationDrawbacks of dynamic programmingTraining your agent with Monte Carlo methodsMonte Carlo predictionMonte Carlo controlTemporal-difference learningOne-step TD learning: TD(0)n-step TD LearningUnderstanding the importance of the simulation in reinforcement learningSummaryReferences
Section 2: Deep Reinforcement Learning
Chapter 6: Deep Q-Learning at Scale
From tabular Q-learning to deep Q-learningNeural Fitted Q-iterationOnline Q-learningDeep Q-networksKey concepts in deep Q-networksThe DQN algorithmExtensions to DQN: RainbowThe extensionsThe performance of the integrated agentHow to choose which extensions to use: Ablations to RainbowOvercoming the deadly triadDistributed deep Q-learningComponents of a distributed deep Q-learning architectureGorila: General reinforcement learning architectureApe-X: Distributed prioritized experience replayImplementing scalable deep Q-learning algorithms using RayA primer on RayRay implementation of a DQN variateRLlib: Production-grade deep reinforcement learningSummaryReferences

Chapter 7: Policy-Based Methods
Need for policy-based methodsA more principled approachAbility to use with continuous action spacesAbility to learn truly random stochastic policiesVanilla policy gradientObjective in the policy gradient methodsFiguring out the gradientREINFORCEThe problem with REINFORCE and all policy gradient methodsVanilla policy gradient using RLlibActor-critic methodsFurther reducing the variance in policy-based methods Advantage Actor-Critic: A2CAsynchronous Advantage Actor-Critic: A3CGeneralized Advantage EstimatorsTrust-region methodsPolicy gradient as policy iterationTRPO: Trust Region Policy OptimizationPPO: Proximal Policy OptimizationRevisiting off-policy MethodsDDPG: Deep Deterministic Policy GradientTD3: Twin Delayed Deep Deterministic Policy GradientSAC: Soft actor-criticIMPALA: Importance Weighted Actor-Learner ArchitectureComparison of the policy-based methods in Lunar LanderHow to pick the right algorithm?Open source implementations of policy-gradient methodsSummaryReferences
Chapter 8: Model-Based Methods
Introducing model-based methodsPlanning through a modelDefining the optimal control problemRandom shootingCross-entropy methodCovariance matrix adaptation evolution strategyMonte Carlo tree searchLearning a world modelUnderstanding what model meansIdentifying when to learn a modelIntroducing a general procedure to learn a modelUnderstanding and mitigating the impact of model uncertainty Learning a model from complex observations Unifying model-based and model-free approaches Refresher on Q-learningDyna-style acceleration of model-free methods using world modelsSummaryReferences
Chapter 9: Multi-Agent Reinforcement Learning
Introducing multi-agent reinforcement learningCollaboration and competition between MARL agentsExploring the challenges in multi-agent reinforcement learningNon-stationarity Scalability Unclear reinforcement learning objective Information sharing Training policies in multi-agent settingsRLlib multi-agent environmentCompetitive self-playTraining tic-tac-toe agents through self-playDesigning the multi-agent tic-tac-toe environmentConfiguring the trainerObserving the resultsSummaryReferences
Section 3: Advanced Topics in RL
Chapter 10: Introducing Machine Teaching
Introduction to machine teachingUnderstanding the need for machine teachingExploring the elements of machine teachingEngineering the reward functionWhen to engineer the reward function Reward shapingExample: Reward shaping for mountain carChallenges with engineering the reward functionCurriculum learningWarm starts with demonstrationsAction maskingSummaryReferences
Chapter 11: Achieving Generalization and Overcoming Partial Observability
Focusing on generalization in reinforcement learningGeneralization and overfitting in supervised learningGeneralization and overfitting in reinforcement learningConnection between generalization and partial observabilityAchieving generalization with domain randomizationOvercoming partial observability with memoryRecipe for generalizationEnriching agent experience via domain randomizationDimensions of randomizationCurriculum learning for generalizationUsing memory to overcome partial observabilityStacking observationsUsing RNNsTransformer architectureQuantifying generalization via CoinRunCoinRun environmentInstalling the CoinRun environmentThe effect of regularization and network architecture on the generalization of RL policiesNetwork Randomization and Feature MatchingSunblaze environmentSummaryReferences
Chapter 12: Meta-Reinforcement Learning
Introducing meta-reinforcement learningLearning to learnDefining meta-reinforcement learning Relation to animal learningRelation to partial observability and domain randomizationMeta-reinforcement learning with recurrent policiesGrid world exampleRLlib implementationGradient-based meta-reinforcement learningRLlib implementationMeta-reinforcement learning as partially observed reinforcement learningChallenges in meta-reinforcement learningConclusionReferences
Chapter 13: Exploring Advanced Topics
Diving deeper into distributed reinforcement learningScalable, efficient deep reinforcement learning: SEED RLRecurrent experience replay in distributed reinforcement learning Experimenting with SEED RL and R2D2Exploring curiosity-driven reinforcement learningCuriosity-driven learning for hard-exploration problemsChallenges in curiosity-driven reinforcement learningNever Give UpAgent57 improvementsOffline reinforcement learningAn overview of how offline reinforcement learning worksWhy we need special algorithms for offline learningWhy offline reinforcement learning is crucialAdvantage weighted actor-criticOffline reinforcement learning benchmarksSummaryReferences
Section 4: Applications of RL
Chapter 14: Solving Robot Learning
Introducing PyBulletSetting up PyBulletGetting familiar with the Kuka environmentGrasping a rectangle block using a Kuka robotKuka Gym environmentDeveloping strategies to solve the Kuka environment Parametrizing the difficulty of the problemUsing curriculum learning to train the Kuka robotCustomizing the environment for curriculum learningDesigning the lessons in the curriculum Training the agent using a manually designed curriculumCurriculum learning using absolute learning progressComparing the experiment resultsGoing beyond PyBullet into autonomous drivingSummaryReferences
Chapter 15: Supply Chain Management
Optimizing inventory procurement decisions The need for inventory and the trade off in its managementComponents of an inventory optimization problemSingle-step inventory optimization: The newsvendor problemSimulating multi-step inventory dynamicsDeveloping a near-optimal benchmark policyReinforcement learning solution to the inventory managementModeling routing problemsPick-up and delivery of online meal ordersPointer networks for dynamic combinatorial optimizationSummaryReferences
Chapter 16: Personalization, Marketing, and Finance
Going beyond bandits for personalizationShortcomings of bandit modelsDeep reinforcement learning for news recommendationDeveloping effective marketing strategies using reinforcement learningPersonalized marketing contentMarketing resource allocation for customer acquisitionReducing customer churn rateWinning back lost customersApplying reinforcement learning in financeChallenges with using reinforcement learning in financeIntroducing TensorTradeDeveloping equity trading strategiesSummaryReferences
Chapter 17: Smart City and Cybersecurity
Controlling traffic lights to optimize vehicle flowIntroducing FlowCreating an experiment in FlowModeling the traffic light control problemSolving the traffic control problem using RLlibFurther readingProviding ancillary service to power gridPower grid operations and ancillary servicesDescribing the environment and the decision-making problemReinforcement learning modelDetecting cyberattacks in a smart gridThe problem of early detection of cyberattacks in a power gridPartial observability of the grid stateSummaryReferences
Chapter 18: Challenges and Future Directions in Reinforcement Learning
What you have achieved with this bookChallenges and future directionsSample efficiencyNeed for high-fidelity and fast simulation modelsHigh-dimensional action spacesReward function fidelitySafety, behavior guarantees, and explainabilityReproducibility and sensitivity to hyper-parameter choicesRobustness and adversarial agentsSuggestions for aspiring reinforcement learning expertsGo deeper into the theoryFollow good practitioners and research labsLearn from papers and from their good explanationsStay up to date with trends in other fields of deep learningRead open source repositoriesPractice!Final wordsReferences
Other Books You May Enjoy
Leave a review - let other readers know what you think

Content preview from Mastering Reinforcement Learning with Python

Chapter 11: Achieving Generalization and Overcoming Partial Observability

Deep reinforcement learning (RL) has achieved what was impossible with the earlier AI methods, such as beating world champions in games like Go, Dota 2, and StarCraft II. Yet, applying RL to real-world problems is still challenging. Two important obstacles to this end are generalization of trained policies to a broad set of environment conditions and developing policies that can handle partial observability. As we will see in the chapter, these are closely related challenges, for which we will present solution approaches.

Here is what we will cover in this chapter:

Focusing on generalization in reinforcement learning
Enriching agent experience via domain randomization ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Hands-On Reinforcement Learning with Python

Publisher Resources

ISBN: 9781838644147

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Mastering Reinforcement Learning with Python

by Enes Bilgin

Chapter 11: Achieving Generalization and Overcoming Partial Observability

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.