Practical Simulations for Machine Learning

Practical Simulations for Machine Learning

by Paris Buttfield-Addison, Mars Buttfield-Addison, Tim Nugent, Jon Manning
Released June 2022
Publisher(s): O'Reilly Media, Inc.
ISBN: 9781492089926

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Buy on Amazon Buy on ebooks.com
Start your free trial

Book description

Simulation and synthesis are core parts of the future of AI and machine learning. Consider: programmers, data scientists, and machine learning engineers can create the brain of a self-driving car without the car. Rather than use information from the real world, you can synthesize artificial data using simulations to train traditional machine learning models.Thatâ??s just the beginning.

With this practical book, youâ??ll explore the possibilities of simulation- and synthesis-based machine learning and AI, concentrating on deep reinforcement learning and imitation learning techniques. AI and ML are increasingly data driven, and simulations are a powerful, engaging way to unlock their full potential.

You'll learn how to:

  • Design an approach for solving ML and AI problems using simulations with the Unity engine
  • Use a game engine to synthesize images for use as training data
  • Create simulation environments designed for training deep reinforcement learning and imitation learning models
  • Use and apply efficient general-purpose algorithms for simulation-based ML, such as proximal policy optimization
  • Train a variety of ML models using different approaches
  • Enable ML tools to work with industry-standard game development tools, using PyTorch, and the Unity ML-Agents and Perception Toolkits

Table of contents

  1. Preface
    1. Resources Used in This Book
    2. Audience and Approach
    3. Organization of This Book
    4. Using This Book
      1. Our Tasks
    5. Conventions Used in This Book
    6. Using Code Examples
    7. O’Reilly Online Learning
    8. How to Contact Us
    9. Acknowledgments
  2. I. The Basics of Simulation and Synthesis
  3. 1. Introducing Synthesis and Simulation
    1. A Whole New World of ML
    2. The Domains
      1. Simulation
      2. Synthesis
    3. The Tools
      1. Unity
      2. PyTorch via Unity ML-Agents
      3. Unity ML-Agents Toolkit
      4. Unity Perception
    4. The Techniques
      1. Reinforcement Learning
      2. Imitation Learning
      3. Hybrid Learning
      4. Summary of Techniques
    5. Projects
      1. Simulation Projects
      2. Synthesis Projects
    6. Summary and Next Steps
  4. 2. Creating Your First Simulation
    1. Everybody Remembers Their First Simulation
    2. Our Simulation
    3. Setting Up
    4. Creating the Unity Project
    5. Packages All the Way Down
    6. The Environment
      1. The Floor
      2. The Target
    7. The Agent
      1. Starting and Stopping the Agent
      2. Letting the Agent Observe the Environment
      3. Letting the Agent Take Actions in the Environment
      4. Giving the Agent Rewards for Its Behavior
      5. Finishing Touches for the Agent
      6. Providing a Manual Control System for the Agent
    8. Training with the Simulation
      1. Monitoring the Training with TensorBoard
      2. When the Training Is Complete
    9. What’s It All Mean?
    10. Coming Up Next
  5. 3. Creating Your First Synthesized Data
    1. Unity Perception
    2. The Process
      1. Using Unity Perception
    3. Creating the Unity Project
    4. Creating a Scene
      1. Getting the Dice Models
      2. A Very Simple Scene
    5. Preparing for Synthesis
      1. Testing the Scenario
      2. Setting Up Our Labels
      3. Checking the Labels
    6. What’s Next?
  6. II. Simulating Worlds for Fun and Profit
  7. 4. Creating a More Advanced Simulation
    1. Setting Up the Block Pusher
    2. Creating the Unity Project
    3. The Environment
      1. The Floor
      2. The Walls
      3. The Block
      4. The Goal
      5. The Agent
      6. The Environment
    4. Training and Testing
  8. 5. Creating a Self-Driving Car
    1. Creating the Environment
      1. The Track
      2. The Car
      3. Setting Up for ML
    2. Training the Simulation
      1. Training
      2. When the Training Is Complete
  9. 6. Introducing Imitation Learning
    1. Simulation Environment
      1. Creating the Ground
      2. Creating the Goal
      3. The Name’s Ball, Agent Ball
      4. The Camera
    2. Building the Simulation
      1. Agent Components
      2. Adding Heuristic Controls
      3. Observations and Goals
    3. Generating Data and Training
      1. Creating Training Data
      2. Configuring for Training
      3. Begin Training
      4. Running with Our Trained Model
    4. Understanding and Using Imitation Learning
  10. 7. Advanced Imitation Learning
    1. Meet GAIL
    2. Do What I Say and Do
      1. A GAIL Scenario
      2. Modifying the Agent’s Actions
      3. Modifying the Observations
      4. Resetting the Agent
      5. Updating the Agent Properties
      6. Demonstration Time
    3. Training with GAIL
    4. Running It and Beyond
  11. 8. Introducing Curriculum Learning
    1. Curriculum Learning in ML
    2. A Curriculum Learning Scenario
    3. Building in Unity
      1. Creating the Ground
      2. Creating the Target
      3. The Agent
    4. Building the Simulation
      1. Making the Agent an Agent
      2. Actions
      3. Observations
      4. Heuristic Controls for Humans
    5. Creating the Curriculum
      1. Resetting the Environment
      2. Curriculum Config
    6. Training
    7. Running It
    8. Curriculum Versus Other Approaches
    9. What’s Next?
  12. 9. Cooperative Learning
    1. A Simulation for Cooperation
      1. Building the Environment in Unity
      2. Coding the Agents
      3. Coding the Environment Manager
      4. Coding the Blocks
      5. Finalizing the Environment and Agents
    2. Training for Cooperation
    3. Cooperative Agents or One Big Agent
  13. 10. Using Cameras in Simulations
    1. Observations and Camera Sensors
    2. Building a Camera-Only Agent
      1. Coding the Camera-Only Agent
      2. Adding a New Camera for the Agent
      3. Seeing What the Agent’s Camera Sees
    3. Training the Camera-Based Agent
    4. Cameras and You
  14. 11. Working with Python
    1. Python All the Way Down
    2. Experimenting with an Environment
    3. What Can Be Done with Python?
      1. Using Your Own Environment
      2. Completely Custom Training
    4. What’s the Point of Python?
  15. 12. Under the Hood and Beyond
    1. Hyperparameters (and Just Parameters)
      1. Parameters
      2. Reward Parameters
      3. Hyperparameters
    2. Algorithms
    3. Unity Inference Engine and Integrations
      1. Using the ML-Agents Gym Wrapper
      2. Side Channels
  16. III. Synthetic Data, Real Results
  17. 13. Creating More Advanced Synthesized Data
    1. Adding Random Elements to the Scene
      1. Randomizing the Floor Color
      2. Randomizing the Camera Position
    2. What’s Next?
  18. 14. Synthetic Shopping
    1. Creating the Unity Environment
    2. A Perception Camera
    3. Faking It Until You Make It
    4. Using Synthesized Data
  19. Index
  20. About the Authors

Product information

  • Title: Practical Simulations for Machine Learning
  • Author(s): Paris Buttfield-Addison, Mars Buttfield-Addison, Tim Nugent, Jon Manning
  • Release date: June 2022
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781492089926