Lemma Let λt ≥ 0 be a step size satisfying


The Q-learning algorithm in Equation (7.22) converges almost surely to the value.

7.5.3     Connection to Differential Dynamic Programming

Q-learning is a technique used to compute an optimal policy for a controlled Markov chain based on observations of the system controlled using a non-optimal policy. Many interesting results have been obtained for models with finite state and action space. Recently, [111] establishes connections between Q-learning and nonlinear control of continuous-time models with general pay-off functions. The authors show that the Hamiltonian appearing in nonlinear control theory is essentially the same as the Q-function that is the object of interest in ...

Get Distributed Strategic Learning for Wireless Engineers now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.