Reinforcement Learning Algorithms with Python

Computing the gradient

As long as the policy is differentiable, its gradient can be easily computed, taking advantage of modern automatic differentiation software.

To do that in TensorFlow, we can define the computational graph and call tf.gradient(loss_function,variables) to calculate the gradient of the loss function (loss_function) with respect to the variables trainable parameters. An alternative would be to directly maximize the objective function using the stochastic gradient descent optimizer, for example, by calling tf.train.AdamOptimizer(lr).minimize(-objective_function).

The following snippet is an example of the steps that are required to compute the approximation in formula (6.5), with a policy of discrete action space of the ...