TensorFlow Reinforcement Learning Quick Start Guide

TensorFlow Reinforcement Learning Quick Start Guide

by Kaushik Balakrishnan
Released March 2019
Publisher(s): Packt Publishing
ISBN: 9781789533583

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Leverage the power of Tensorflow to Create powerful software agents that can self-learn to perform real-world tasks

Key Features

  • Explore efficient Reinforcement Learning algorithms and code them using TensorFlow and Python
  • Train Reinforcement Learning agents for problems, ranging from computer games to autonomous driving.
  • Formulate and devise selective algorithms and techniques in your applications in no time.

Book Description

Advances in reinforcement learning algorithms have made it possible to use them for optimal control in several different industrial applications. With this book, you will apply Reinforcement Learning to a range of problems, from computer games to autonomous driving.

The book starts by introducing you to essential Reinforcement Learning concepts such as agents, environments, rewards, and advantage functions. You will also master the distinctions between on-policy and off-policy algorithms, as well as model-free and model-based algorithms. You will also learn about several Reinforcement Learning algorithms, such as SARSA, Deep Q-Networks (DQN), Deep Deterministic Policy Gradients (DDPG), Asynchronous Advantage Actor-Critic (A3C), Trust Region Policy Optimization (TRPO), and Proximal Policy Optimization (PPO). The book will also show you how to code these algorithms in TensorFlow and Python and apply them to solve computer games from OpenAI Gym. Finally, you will also learn how to train a car to drive autonomously in the Torcs racing car simulator.

By the end of the book, you will be able to design, build, train, and evaluate feed-forward neural networks and convolutional neural networks. You will also have mastered coding state-of-the-art algorithms and also training agents for various control problems.

What you will learn

  • Understand the theory and concepts behind modern Reinforcement Learning algorithms
  • Code state-of-the-art Reinforcement Learning algorithms with discrete or continuous actions
  • Develop Reinforcement Learning algorithms and apply them to training agents to play computer games
  • Explore DQN, DDQN, and Dueling architectures to play Atari's Breakout using TensorFlow
  • Use A3C to play CartPole and LunarLander
  • Train an agent to drive a car autonomously in a simulator

Who this book is for

Data scientists and AI developers who wish to quickly get started with training effective reinforcement learning models in TensorFlow will find this book very useful. Prior knowledge of machine learning and deep learning concepts (as well as exposure to Python programming) will be useful.

Table of contents

  1. Title Page
  2. Copyright and Credits
    1. TensorFlow Reinforcement Learning Quick Start Guide
  3. Dedication
  4. About Packt
    1. Why subscribe?
    2. Packt.com
  5. Contributors
    1. About the author
    2. About the reviewer
    3. Packt is searching for authors like you
  6. Preface
    1. Who this book is for
    2. What this book covers
    3. To get the most out of this book
      1. Download the example code files
      2. Download the color images
      3. Conventions used
    4. Get in touch
      1. Reviews
  7. Up and Running with Reinforcement Learning
    1. Why RL?
      1. Formulating the RL problem
    2. The relationship between an agent and its environment
      1. Defining the states of the agent
      2. Defining the actions of the agent
      3. Understanding policy, value, and advantage functions
    3. Identifying episodes
    4. Identifying reward functions and the concept of discounted rewards
      1. Rewards
    5. Learning the Markov decision process
    6. Defining the Bellman equation
    7. On-policy versus off-policy learning
      1. On-policy method
      2. Off-policy method
    8. Model-free and model-based training
    9. Algorithms covered in this book
    10. Summary
    11. Questions
    12. Further reading
  8. Temporal Difference, SARSA, and Q-Learning
    1. Technical requirements
    2. Understanding TD learning
      1. Relation between the value functions and state
    3. Understanding SARSA and Q-Learning
      1. Learning SARSA
      2. Understanding Q-learning
    4. Cliff walking and grid world problems
      1. Cliff walking with SARSA
      2. Cliff walking with Q-learning
      3. Grid world with SARSA
    5. Summary
    6. Further reading
  9. Deep Q-Network
    1. Technical requirements
    2. Learning the theory behind a DQN
    3. Understanding target networks
    4. Learning about replay buffer
    5. Getting introduced to the Atari environment
      1. Summary of Atari games
        1. Pong
        2. Breakout
        3. Space Invaders
        4. LunarLander
        5. The Arcade Learning Environment
      2. Coding a DQN in TensorFlow
        1. Using the model.py file
        2. Using the funcs.py file
        3. Using the dqn.py file
      3. Evaluating the performance of the DQN on Atari Breakout
    6. Summary
    7. Questions
    8. Further reading
  10. Double DQN, Dueling Architectures, and Rainbow
    1. Technical requirements
    2. Understanding Double DQN
      1. Updating the Bellman equation
      2. Coding DDQN and training to play Atari Breakout
      3. Evaluating the performance of DDQN on Atari Breakout
    3. Understanding dueling network architectures
      1. Coding dueling network architecture and training it to play Atari Breakout
        1. Combining V and A to obtain Q
      2. Evaluating the performance of dueling architectures on Atari Breakout
    4. Understanding Rainbow networks
      1. DQN improvements
        1. Prioritized experience replay
        2. Multi-step learning
        3. Distributional RL
        4. Noisy nets
    5. Running a Rainbow network on Dopamine
      1. Rainbow using Dopamine
    6. Summary
    7. Questions
    8. Further reading
  11. Deep Deterministic Policy Gradient
    1. Technical requirements
    2. Actor-Critic algorithms and policy gradients
      1. Policy gradient
    3. Deep Deterministic Policy Gradient
      1. Coding ddpg.py
      2. Coding AandC.py
      3. Coding TrainOrTest.py
      4. Coding replay_buffer.py
    4. Training and testing the DDPG on Pendulum-v0
    5. Summary
    6. Questions
    7. Further reading
  12. Asynchronous Methods - A3C and A2C
    1. Technical requirements
    2. The A3C algorithm
      1. Loss functions
      2. CartPole and LunarLander
        1. CartPole
        2. LunarLander
    3. The A3C algorithm applied to CartPole
      1. Coding cartpole.py
      2. Coding a3c.py
        1. The AC class
      3. The Worker() class
      4. Coding utils.py
      5. Training on CartPole
    4. The A3C algorithm applied to LunarLander
      1. Coding lunar.py
      2. Training on LunarLander
    5. The A2C algorithm
    6. Summary
    7. Questions
    8. Further reading
  13. Trust Region Policy Optimization and Proximal Policy Optimization
    1. Technical requirements
    2. Learning TRPO
      1. TRPO equations
    3. Learning PPO
      1. PPO loss functions
    4. Using PPO to solve the MountainCar problem
      1. Coding the class_ppo.py file
      2. Coding train_test.py file
    5. Evaluating the performance
      1. Full throttle
      2. Random throttle
    6. Summary
    7. Questions
    8. Further reading
  14. Deep RL Applied to Autonomous Driving
    1. Technical requirements
    2. Car driving simulators
    3. Learning to use TORCS
      1. State space
      2. Support files
    4. Training a DDPG agent to learn to drive
      1. Coding ddpg.py
      2. Coding AandC.py
      3. Coding TrainOrTest.py
    5. Training a PPO agent
    6. Summary
    7. Questions
    8. Further reading
  15. Assessment
    1. Chapter 1
    2. Chapter 3
    3. Chapter 4
    4. Chapter 5
    5. Chapter 6
    6. Chapter 7
    7. Chapter 8
  16. Other Books You May Enjoy
    1. Leave a review - let other readers know what you think

Product information

  • Title: TensorFlow Reinforcement Learning Quick Start Guide
  • Author(s): Kaushik Balakrishnan
  • Release date: March 2019
  • Publisher(s): Packt Publishing
  • ISBN: 9781789533583