Practical Simulations for Machine Learning

Chapter 7. Advanced Imitation Learning

In this chapter, we’re going to look at imitation learning (IL) using generative adversarial imitation learning (GAIL). We could use GAIL in an almost identical fashion to what we did when we used IL for behavioral cloning (BC), but that wouldn’t really be showing you anything new other than changing the configuration YAML file.

With our simulations so far, we’ve done the basics, built upon them, and created a simple self-driving car, all using reinforcement learning. And in the previous chapter, we used IL to train an agent using human behavior. The IL we used for behavioral cloning attempted to maximize its similarity to our provided training data.

IL is not the only BC technique we can use. This time, we’ll use GAIL. GAIL can help improve the training of our agent, allowing it to essentially jump over the early hurdles in the learning process and let it focus on improving itself from then on.

Tip

BC and GAIL can also be combined so that you can hopefully extract the benefits of both and mitigate the weaknesses of either. Toward the end of this chapter, we’ll cover how you can combine GAIL and BC, but for now the focus will be on GAIL.

Meet GAIL

Before we start working on a GAIL-based activity with Unity and ML-Agents, we’re going to unpack a little bit of what makes GAIL tick.

GAIL is, as its name implies, an adversarial approach to imitation learning and is based on a type of machine learning network called a GAN: a generative adversarial ...