Reinforcement Learning and Stochastic Optimization

3 Online Learning

There is a massive community that has evolved under names such as statistics, statistical learning, machine learning, and data sciences. The vast majority of this work, known as supervised learning, involves taking a dataset (xⁿ, yⁿ), n = 1, …, N of input data xⁿ and corresponding observations (sometimes called “labels”) yⁿ and using this to design a statistical model f(x|θ) that produces the best fit between f(x|θ) and the associated observation (or label) yⁿ. This is the world of big data.

This book is on the topic of making decisions (that we call x). So why do we need a chapter on learning? The simple explanation is that machine learning arises throughout the process of helping computers make decisions. Classical machine learning is focused on learning something about an exogenous process: forecasting weather, predicting demand, estimating the performance of a drug or material. In this book, we need exogenous learning for the same reason everyone else does, but most of the time we will focus on endogenous learning, where we are learning about value functions, policies, and response surfaces, which are learning problems that arise in the context of methods for making decisions.

We open this chapter with an overview of the role of machine learning in the context of sequential decision making. The remainder of the chapter is an introduction to machine learning, with an emphasis on learning over time, a topic known as online learning, since this will dominate ...