In this example, we want to employ an alternative algorithm called Actor-Critic, together with TD(0). In this method, the agent is split into two components, a Critic, which is responsible for evaluating the quality of the value estimation, and an actor, which selects and performs an action. As pointed out by Dayan (in Theoretical Neuroscience, Dayan P., Abbott L. F., The MIT Press), the dynamics of an Actor-Critic approach are similar to the interleaving policy evaluation and policy improvement steps. In fact, the knowledge of the Critic is obtained through an iterative process, and its initial evaluations are normally sub-optimal.
The structural schema is shown in the following diagram: ...
