Optimization for Learning and Control

8Dynamic Programming

In Chapter 7, we discussed how to solve optimal control problems over a finite time interval. We specifically considered the continuous time case, since for discrete time dynamics, it is fairly straightforward how to solve the optimal control problem. The solutions we obtained are so‐called open‐loop solutions, i.e. the value of the control signal at a certain time depends only on the initial value of the state and not on the current value of the state. For the linear quadratic (LQ) control problem, we were able to restate the solution as a feedback policy, i.e. to explicitly write the control signal as for a feedback function . This is very desirable since it is known that feedback solutions are more robust to unmodeled dynamics and disturbances. We will in this chapter look in more detail into the problem of obtaining feedback solutions. We will only treat the discrete time case, since it is easier from a mathematical point of view. However, many of the ideas can be extended to the continuous time case. We will first consider a finite time interval and then we will discuss the case of an infinite time interval. Optimal feedback control goes back to the work by Richard Bellman who in 1953 introduced what is known as dynamic programming to solve these types ...