Reinforcement Learning

Appendix A. The Gradient of a Logistic Policy for Two Actions

Equation 5-6 is a policy for two actions. To update the policies I need to calculate the gradient of the natural logarithm of the policy (see Equation 5-4). I present this in Equation A-1. You can perform the differentiation in a few different ways depending on how you refactor it, so the result can look different, even though it provides the same result.

Equation A-1. Logistic policy gradient for two actions

$$\nabla \ln \pi (a\mid s,\theta )=\left(\begin{array}{c}\frac{\delta }{\delta {\theta }_0}\ln \left(\frac{1}{1+e^{-{\theta }_0^{⊺}s}}\right)\\ \frac{\delta }{\delta {\theta }_1}\ln (1-\frac{1}{1+e^{-{\theta }_1^{⊺}s}})\end{array}\right)$$

I calculate the gradients of each action independently and I find it easier if I refactor the logistic function like in Equation A-2.

Equation A-2. Refactoring the logistic function

$$\begin{array}{cc}\hfill \pi (x)\doteq \frac{1}{1+e^{-x}}& =\frac{e^x}{e^x(1+e^{-x})}\hfill \\ =\frac{e^x}{e^x(1+e^{-x})}\hfill \\ =\frac{e^x}{e^x+e^x{e}^{-x}}\hfill \\ =\frac{e^x}{e^x+e^{x-x}}\hfill \\ =\frac{e^x}{e^x+e^0}\hfill \\ =\frac{e^x}{e^x+1}\hfill \\ =\frac{e^x}{1+e^x}\hfill \end{array}$$

The derivative of the refactored logistic function, for action 0, is shown in Equation A-3.

Equation A-3. Differentiation of action 0

$$\begin{array}{cc}\hfill \frac{\delta }{\delta {\theta }_0}\ln {\pi }_0({\theta }_0^{⊺}s)& =\frac{\delta }{\delta {\theta }_0}\ln \left(\frac{e^{{\theta }_0^{⊺}s}}{1+e^{{\theta }_0^{⊺}s}}\right)\hfill \\ =\frac{\delta }{\delta {\theta }_0}\ln e^{{\theta }_0^{⊺}s}-\frac{\delta }{\delta {\theta }_0}\ln (1+e^{{\theta }_0^{⊺}s})\hfill \\ =\frac{\delta }{\delta {\theta }_0}{\theta }_0^{⊺}s-\frac{\delta }{\delta {\theta }_0}\ln (1+e^{{\theta }_0^{⊺}s})\hfill \\ =s-\frac{\delta }{\delta {\theta }_0}\ln (1+e^{{\theta }_0^{⊺}s})\hfill \\ =s-\frac{\delta }{\delta {\theta }_0}\ln u\text{where }u=1+e^{{\theta }_0^{⊺}s}\hfill \\ =s-\frac{1}{u}\frac{\delta }{\delta {\theta }_0}u\hfill \\ =s-\frac{1}{1+e^{{\theta }_0^{⊺}s}}\frac{\delta }{\delta {\theta }_0}(1+e^{{\theta }_0^{⊺}s})\hfill \\ =s-\frac{1}{1+e^{{\theta }_0^{⊺}s}}\frac{\delta }{\delta v}(e^v)\frac{\delta }{\delta {\theta }_0}v\text{where }v={\theta }_0^{⊺}s\hfill \\ =s-\frac{1}{1+e^{{\theta }_0^{⊺}s}}{e}^{{\theta }_0^{⊺}s}s\hfill \\ =s-\frac{s{e}^{{\theta }_0^{⊺}}s}{1+e^{{\theta }_0^{⊺}s}}\hfill \\ =s-s\pi ({\theta }_0^{⊺}s)\hfill \end{array}$$

The method to calculate the derivative of the policy for action 1 is shown in Equation A-4. Note that the derivation towards the end is the same as Equation A-3.

Equation A-4. Differentiation of action 1

$$\begin{array}{cc}\hfill \frac{\delta }{\delta {\theta }_1}\ln {\pi }_1({\theta }_1^{⊺}s)& =\frac{\delta }{\delta {\theta }_1}\ln (1-\frac{e^{{\theta }_1^{⊺}s}}{1+e^{{\theta }_1^{⊺}s}})\hfill \\ =\frac{\delta }{\delta {\theta }_1}\ln (\frac{1+e^{{\theta }_1^{⊺}s}}{1+e^{{\theta }_1^{⊺}s}}-\frac{e^{{\theta }_1^{⊺}s}}{1+e^{{\theta }_1^{⊺}s}})\hfill \\ =\frac{\delta }{\delta {\theta }_1}\ln \left(\frac{1+e^{{\theta }_1^{⊺}s}-e^{{\theta }_1^{⊺}s}}{1+e^{{\theta }_1^{⊺}s}}\right)\hfill \\ =\frac{\delta }{\delta {\theta }_1}\ln \left(\frac{1}{1+e^{{\theta }_1^{⊺}s}}\right)\hfill \\ =\frac{\delta }{\delta }\hfill \end{array}$$