Hands-On Reinforcement Learning for Games

Trust region methods

A further optimization we can apply to gradient ascent is using trust regions or controlled regions of updates. These methods are of course fundamental to TRPO given the name but the concept is further extended to other policy-based methods. In TRPO, we extend regions of trust over the approximation functions using the Minorize-Maximization or MM algorithm. The intuition of MM is that there is a lower bound function that we can always expect the returns/reward to be higher than. Hence, if we maximize this lower bound function, we also attain our best policy. Gradient descent by default is a line search algorithm but this again introduces the problem of overshooting. Instead, we can first approximate the step size and ...