Handbook of Healthcare Analytics

14 Markov Decision Processes

Alan Scheller‐Wolf

Carnegie Mellon University

14.1 Introduction

If we were to create an idealized analytical tool to assist in healthcare decision‐making, we would be hard pressed—at first glance at least—to devise a tool superior to Markov Decision Processes (MDPs). MDPs are particularly well‐suited for the decisions that need to be made within healthcare when there is (are):

• A high degree of uncertainty, which is at least partially resolved over time, possibly in response to actions the decision‐maker takes (i.e., a diagnosis arrived at after a series of tests).
• A well‐defined objective or objectives, such as maximizing a patient's longterm health.
• A fundamentally sequential nature; in other words, the decision‐maker usually has the ability to select from a series of actions (tests, treatments, medicines, procedures) over time in response to different signals received from the patient, in order to pursue the objective.
• Expectations about future signals and actions. Crucially, when choosing an action in the present, the decision‐maker must also consider issues such as her future budget, the future possible course of the condition, future potential treatments in response to the future course of the condition, and/or future side effects of current or future treatments.

If we were to get a little greedy, we might design this idealized tool to not only be able to provide the desired course of action (and reaction) to the evolution of the patient ...