Reinforcement Learning

Appendix B. The Gradient of a Softmax Policy

The derivation of the gradient of a softmax policy is shown in Equation B-1. Note how this is a similar form to the logistic gradients in Appendix A.

Equation B-1. Gradient of a softmax policy

$$\begin{array}{cc}\hfill {\nabla }_{\theta }\ln \pi ({\theta }^{⊺}s)& ={\nabla }_{\theta }\ln \frac{e^{{\theta }^{⊺}}s}{{\sum }_a{e}^{{\theta }_a^{⊺}s}}\hfill \\ ={\nabla }_{\theta }\ln e^{{\theta }^{⊺}}s-{\nabla }_{\theta }\ln {\sum }_a{e}^{{\theta }_a^{⊺}s}\hfill \\ ={\nabla }_{\theta }{\theta }^{⊺}s-{\nabla }_{\theta }\ln {\sum }_a{e}^{{\theta }_a^{⊺}s}\hfill \\ =s-{\nabla }_{\theta }\ln {\sum }_a{e}^{{\theta }_a^{⊺}s}\hfill \\ =s-{\nabla }_{\theta }\ln {\sum }_a{e}^{{\theta }_a^{⊺}s}\hfill \\ =s-\frac{{\nabla }_{\theta }{\sum }_a{e}^{{\theta }_a^{⊺}s}}{{\sum }_a{e}^{{\theta }_a^{⊺}s}}\hfill \\ =s-\frac{{\sum }_{a}s{e}^{{\theta }_a^{⊺}s}}{{\sum }_a{e}^{{\theta }_a^{⊺}s}}\hfill \\ =s-{\sum }_{a}s\pi ({\theta }^{⊺}s)\hfill \end{array}$$