Hands-On Machine Learning for Algorithmic Trading

Sequences of states, actions, and rewards

MDPs proceed in the following fashion: at each step, t, the agent observes the environment's state, S_t ∈ S, and selects an action, A_t ∈ A, where S and A are the sets of states and actions, respectively. At the next time step, t+1, the agent receives a reward, R_t+1∈ R, and transitions to the state, S_t+1. Over time, the MDP gives rise to a trajectory, S₀, A₀, R₁, S₁, A₁, R₁, ..., that continues until the agent reaches a terminal state and the episode ends.

Finite MDPs with a limited number of actions, A, states, S, and rewards, R, include well-defined discrete probability distributions over these elements. Due to the Markov property, these distributions only depend on the previous state and action. ...