10 Uncertainty Modeling

We cannot find an effective policy unless we are modeling the problem properly. In the realm of sequential decision problems, this means accurately modeling uncertainty. The importance of modeling uncertainty has been underrepresented in the stochastic optimization literature, although practitioners working on real problems have long been aware of both the importance and the challenges of modeling uncertainty.

Fortunately, there is a substantial body of research focused on the modeling of uncertainty and stochastic processes that has evolved in the communities working on Monte Carlo simulation and uncertainty quantification. We use uncertainty modeling as the broader term that describes the process of identifying and modeling uncertainty, while simulation refers to the vast array of tools that break down complex stochastic processes using the computational tools of Monte Carlo simulation.

It helps to remind ourselves of the two information processes that drive any sequential stochastic optimization problem: decisions, and exogenous information. Assume that we can pick some policy Xtπ(St). We need to be able to simulate a sample realization of the policy, which will look like

StartLayout 1st Row upper S 0 right-arrow x 0 equals upper X 0 Superscript pi Baseline left-parenthesis upper S 0 right-parenthesis right-arrow upper W 1 right-arrow upper S 1 right-arrow x 1 equals upper X 1 Superscript pi Baseline left-parenthesis upper S 1 right-parenthesis right-arrow upper W 2 right-arrow upper S 3 right-arrow EndLayout

Given our policy, this simulation assumes that we have access to a transition function


We can execute equation (10.1) if we are given a policy Xtπ(St) and if we have access ...

Get Reinforcement Learning and Stochastic Optimization now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.