Hands-On Reinforcement Learning for Games

Training advantage AC

Using advantage and training multiple networks to work together as you may imagine is not trivial. Therefore, we want to focus a whole exercise on understanding how training works in AC. Open up Chapter_8_ActorCritic.py again and follow the exercise:

1. Our main focus will be the train_net function in the ActorCritic class we saw before. Starting with the first two lines, we can see this is where the training batch is first made and we calculate td_target. Recall we covered the form of TD error calculation check when we implemented DDQN:

s, a, r, s_prime, done = self.make_batch()td_target = r + gamma * self.v(s_prime) * done

2. Next, we calculate the change or delta between our target and the value function. Again, this ...