Reinforcement Learning and Stochastic Optimization

6 Stepsize Policies

There is a wide range of adaptive learning problems that depend on an iteration of the form we first saw in chapter 5 that looks like

  (6.1)

The stochastic gradient ${\nabla }_{x}F(x^n,W^{n+1})$ tells us what direction to go in, but we need the stepsize ${\alpha }_n$ to tell us how far we should move.

There are two important settings where this formula is used. The first is where we are maximizing some metric such as contributions, utility, or performance. In these settings, the units of ${\nabla }_{x}F(x^n,W^{n+1})$ and the decision variable $x$ are different, so the stepsize has to perform the scaling so that the size of ${\alpha }_n{\nabla }_{x}F(x^n,W^{n+1})$ is not too large or too small relative to $x^n$.

A second and very important setting arises in what is known as supervised learning. In this context, we are trying to estimate some function $f(x|\theta )$ using observations $y=f(x|\theta )+\epsilon$. In this context, $f(x|\theta )$ and $y$ have the same scale. We encounter these problems in three settings:

• Approximating the function $\mathbb{E}F(x,W)$ to create an estimate $\overline{F}(x)$ that can be optimized.
• Approximating the value $V_t(S_t)$ of being in a state $S_t$ and then following some policy (we encounter this problem starting in chapters 16 and 17 when we introduce approximate dynamic programming).
• Creating a parameterized policy $X^{\pi }(S|\theta )$ to fit observed decisions. Here, we assume we have access to some method of creating a decision $x$ and then we use this to create a parameterized ...