Statistics for Machine Learning

Markov decision processes and Bellman equations

Markov decision process (MDP) formally describes an environment for reinforcement learning. Where:

• Environment is fully observable
• Current state completely characterizes the process (which means the future state is entirely dependent on the current state rather than historic states or values)
• Almost all RL problems can be formalized as MDPs (for example, optimal control primarily deals with continuous MDPs)

Central idea of MDP: MDP works on the simple Markovian property of a state; for example, S_t+1 is entirely dependent on latest state S_t rather than any historic dependencies. In the following equation, the current state captures all the relevant information from the history, which means ...