Reinforcement Learning and Stochastic Optimization

9 Modeling Sequential Decision Problems

Perhaps one of the most important skills to develop when solving sequential decision problems is the ability to write down a mathematical model of the problem. As illustrated in Figure 9.1, the path from a real application to doing computational work on the computer has to pass through the process of mathematical modeling. Unlike fields such as deterministic optimization and machine learning, there is not a standard modeling framework for decisions under uncertainty. This chapter will develop, in much greater detail, our universal modeling framework for any sequential decision problem. Although we have introduced this framework in earlier chapters, this chapter is dedicated to modeling, bringing out the incredible richness of sequential decision problems. This chapter is written to stand alone, so there is some repetition of elements of our universal model.

Figure 9.1 The path from applications to computation passes through the need for a mathematical model.

While the problem domain of sequential decision problems is astonishingly rich, we can write any sequential decision problem as the sequence:

Let x_t be the decision we make at time $t$, and let $W_{t+1}$ be the new information that arrives between $t$ (that is, after the decision has been made), ...