Practical Simulations for Machine Learning

Practical Simulations for Machine Learning

by Paris Buttfield-Addison, Jon Manning, Mars Buttfield-Addison, Tim Nugent
Released June 2022
Publisher(s): O'Reilly Media, Inc.
ISBN: 9781492089926

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to live online training experiences, plus books, videos, and digital content from O’Reilly and nearly 200 trusted publishing partners.

Start your free trial

Book description

Simulation and synthesis are core parts of the future of AI and machine learning. Consider: programmers, data scientists, and machine learning engineers can create the brain of a self-driving car without the car. Rather than use information from the real world, you can create artificial data using simulations to train traditional machine learning models. That's just the beginning.

With this practical book, you'll explore the possibilities of simulation- and synthesis-based machine learning and AI, with a focus on deep reinforcement learning and imitation learning techniques. AI and ML are increasingly data driven, and simulations are a powerful, engaging way to unlock their full potential.

With this deeply practical book, you'll learn how to:

  • Design an approach for solving ML and AI problems using simulations
  • Use a game engine to synthesize images for use as training data
  • Create simulation environments designed for training deep reinforcement learning and imitation learning
  • Use and apply efficient general-purpose algorithms for simulation-based ML, such as proximal policy optimization (PPO) and soft actor-critic (SAO)
  • Train ML models locally, concurrently, and in the cloud
  • Use PyTorch, TensorFlow, the Unity ML-Agents and Perception Toolkits to enable ML tools to work with industry-standard game development tools

Publisher resources

View/Submit Errata

Table of contents

  1. Preface
    1. Resources Used in This Book
    2. Audience and Approach
    3. Organization of This Book
    4. Using This Book
      1. Our Tasks
    5. Conventions Used in This Book
    6. Using Code Examples
    7. O’Reilly Online Learning
    8. How to Contact Us
    9. Acknowledgments
  2. I. The Basics of Simulation and Synthesis
  3. 1. Introducing Synthesis and Simulation
    1. A Whole New World of ML
    2. The Domains
      1. Simulation
      2. Synthesis
    3. The Tools
      1. Unity
      2. PyTorch via Unity ML-Agents
      3. Unity ML-Agents Toolkit
      4. Unity Perception
    4. The Techniques
      1. Reinforcement Learning
      2. Imitation Learning
      3. Hybrid Learning
      4. Summary of Techniques
    5. Projects
      1. Simulation Projects
      2. Synthesis Projects
    6. Summary and Next Steps
  4. 2. Creating Your First Simulation
    1. Everybody Remembers Their First Simulation
    2. Our Simulation
    3. Setting Up
    4. Creating the Unity Project
    5. Packages All the Way Down
    6. The Environment
      1. The Floor
      2. The Target
    7. The Agent
      1. Starting and Stopping the Agent
      2. Letting the Agent Observe the Environment
      3. Letting the Agent Take Actions in the Environment
      4. Giving the Agent Rewards for its Behavior
      5. Finishing Touches for an Agent
      6. Providing a Manual Control System for the Agent
    8. Training with the Simulation
      1. Monitoring the Training with TensorBoard
      2. When the Training Is Complete
    9. What’s It All Mean?
      1. Coming Up Next
  5. 3. Creating Your First Synthesised Data
    1. Unity Perception
    2. The Process
      1. Using Unity Perception
    3. Creating the Unity Project
    4. Creating a Scene
      1. Getting the Dice Models
      2. A Very Simple Scene
    5. Preparing for Synthesis
      1. Test the Scenario
      2. Setting Up our Labels
      3. What’s next?
  6. II. Simulating Worlds for Fun and Profit
  7. 4. Creating a More Advanced Simulation
    1. Setting Up the Block Pusher
    2. Creating the Unity project
    3. The Environment
      1. The Floor
      2. The Walls
      3. The Block
      4. The Goal
      5. The Agent
      6. The Environment
    4. Training and Testing
  8. 5. Creating a Self-Driving Car
    1. Creating the Environment
      1. The Track
      2. The Car
      3. Setting Up for ML
    2. Training the Simulation
      1. Training
      2. When the Training Is Complete
  9. 6. Introducing Imitation Learning
    1. Simulation Environment
      1. Creating the Ground
      2. Creating the Goal
      3. The Name’s Ball, Agent Ball.
      4. The Camera
    2. Building the Simulation
      1. Agent Components
      2. Adding in Heuristic Controls
      3. Observations and Goals
    3. Generating Data and Training
      1. Creating Training Data
      2. Configuring for Training
      3. Begin Training
      4. Running with our trained model
  10. 7. Advanced Imitation Learning
    1. Meet GAIL
    2. Do What I Say and Do
      1. A GAIL Scenario
      2. Modifying the Agents Actions
      3. Modifying the Observations
      4. Resetting the Agent
      5. Updating the Agent Properties
      6. Demonstration Time
    3. Training with GAIL
      1. Running It and Beyond
  11. 8. Introducing Curriculum Learning
    1. Curriculum Learning in ML
    2. A Curriculum Learning Scenario
    3. Building in Unity
      1. Creating the Ground
      2. Creating the Target
      3. The Agent
    4. Building the Simulation
      1. Making the Agent an Agent
      2. Actions
      3. Observations
      4. Heuristic Controls for Humans
    5. Creating the Curriculum
      1. Resetting the Environment
      2. Curriculum Config
    6. Training
      1. Running It
    7. Curriculum Versus Other Approaches
      1. What’s Next?
  12. 9. Cooperative Learning
    1. A Simulation for Cooperation
      1. Building the Environment in Unity
      2. Coding the Agents
      3. Coding the Environment Manager
      4. Coding the Blocks
      5. Finalizing the Environment and Agents
    2. Training for Cooperation
  13. 10. Using Cameras in Simulations
    1. Observations and Camera Sensors
    2. Building a Camera-Only Agent
      1. Coding the Camera-Only Agent
      2. Adding a New Camera for the Agent
      3. Seeing What the Agent’s Camera Sees
    3. Training the Camera-Based Agent
  14. 11. Working with Python
    1. Python All the Way Down
      1. Experimenting with an Environment
    2. What Can Be Done with Python?
      1. Using Your Own Environment
      2. Completely Custom Training
  15. 12. Under the Hood and Beyond
    1. Hyperparameters (and Just Parameters)
      1. Parameters
      2. Reward Parameters
      3. Hyperparameters
    2. Algorithms
    3. The Unity Inference Engine
    4. Using the ML-Agents Gym Wrapper
      1. Unity Environments and the OpenAI Baselines
    5. Side Channels
      1. Engine Configuration Channel
      2. Environment Parameters Channel
  16. III. Synthetic Data, Real Results
  17. 13. Creating More Advanced Synthesized Data
    1. Adding Random to the Scene
      1. Randomizing the Floor Color
      2. Randomizing the Camera Position
    2. What’s Next
  18. 14. Synthetic Shopping
    1. Creating the Unity Environment
    2. Faking It Until You Make It
    3. Using Synthesized Data
  19. About the Authors

Product information

  • Title: Practical Simulations for Machine Learning
  • Author(s): Paris Buttfield-Addison, Jon Manning, Mars Buttfield-Addison, Tim Nugent
  • Release date: June 2022
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781492089926