Reinforcement Learning

Chapter 7. Learning All Possible Policies with Entropy Methods

Deep reinforcement learning (RL) is a standard tool due to its ability to process and approximate complex observations, which result in elaborate behaviors. However, many deep RL methods optimize for a deterministic policy, since if you had full observability, there is only one best policy. But it is often desirable to learn a stochastic policy or probabilistic behaviors to improve robustness and deal with stochastic environments.

What Is Entropy?

Shannon entropy (abbreviated to entropy from now on) is a measure of the amount of information contained within a stochastic variable, where information is calculated as the number of bits required to encode all possible states. Equation 7-1 shows this as an equation where $X\doteq \{x_0,x_1,\dots ,x_{n-1}\}$ is a stochastic variable, $\mathscr{H}$ is the entropy, $I$ is the information content, and $b$ is the base of the logarithm used (commonly bits for $b\doteq 2$, bans for $b\doteq 10$, and nats for $b\doteq e$). Bits are the most common base.

Equation 7-1. The information content of a random variable

$$\mathscr{H}(X)\doteq 𝔼[I(X)]=-\sum _{x\in X}p(x){\log }_{b}p(x)$$

For example, a coin has two states, assuming it doesn’t land on its edge. These two states can be encoded by a zero and a one, therefore the amount of information contained within a coin, measured by entropy in bits, is one. A die has six possible states, so you would need three bits to describe all of those states (the real value is 2.5849…​).

A probabilistic solution to optimal control is a stochastic ...