Hands-On Machine Learning for Algorithmic Trading

Generalized policy iteration

In practice, there are several ways to truncate policy iteration, for example, by evaluating the policy k times before improving it. This just means that the max operator will only be applied at every k^th iteration.

Most RL algorithms estimate value and policy functions, and rely on the interaction of policy evaluation and improvement to converge to a solution, as illustrated in the following diagram. The general approach improves the policy with respect to the value function, while adjusting the value function to match the policy:

Convergence requires that the value function is consistent with the policy, which ...