Hands-On Machine Learning for Algorithmic Trading

From a value function to an optimal policy

The solution to an RL problem is a policy that optimizes the cumulative reward. Policies and value functions are closely connected: an optimal policy yields a value estimate for each state, v_π (s), or state-action pair, q_π (s, a), that is at least as high as for any other policy since the value is the cumulative reward under the given policy. Hence, the optimal value functions, v * (s) = max_π v_π (s) and q * (s, a) = max_π q_π (s, a), implicitly define optimal policies and solve the MDP.

The optimal value functions, v * and q *, also satisfy the Bellman equations from the previous section. These Bellman optimality equations can omit the explicit reference to a policy as it is implied by v * and q*