Reinforcement Learning Algorithms with Python

Implementing REINFORCE with baseline

The value function that baseline approximated with a neural network can be implemented by adding a few lines to our previous code:

1. Add the neural network, the operations for computing the MSE loss function, and the optimization procedure to the computational graph:

    ...    # placeholder that will contain the reward to go values (i.e. the y values)    rtg_ph = tf.placeholder(shape=(None,), dtype=tf.float32, name='rtg')        # MLP value function    s_values = tf.squeeze(mlp(obs_ph, hidden_sizes, 1, activation=tf.tanh))    # MSE loss function    v_loss = tf.reduce_mean((rtg_ph - s_values)**2)    # value function optimization    v_opt = tf.train.AdamOptimizer(vf_lr).minimize(v_loss)    ...

2. Run s_values, and store the predictions, ...