Introduction to Stochastic Models

Chapter 3

Elements of Markov Modeling

This chapter is devoted to the study of Markov models used in several problems of physics, biology, economy and engineering.

3.1. Markov models: ideas, history, applications

The concept of Markovian dependence that we owe to the Russian mathematician A. A. Markov (1856-1922) appeared for the first time in an explicit form in the article “Extension of the law of large numbers to quantities dependent on each other” (in Russian).¹ Later, Markov studied the properties of some sequences of dependent r.v. that nowadays are called Markov chains. He wanted to generalize classic properties of independent r.v. to sequences of r.v. for which the independence hypothesis is not satisfied. On the one hand, it is clear that if the concept of dependence had been too general, then the extension of the properties of independent r.v. would have been either impossible or restricted to a limited number of results. On the other hand, the notion of dependence that we consider needs to be “natural” in the sense that it has to really be encountered in a certain range of applications. The concept of Markovian dependence has the quality of complying with these criteria.

In the same period, the French mathematician H. Poincaré, studying the problem of shuffling playing cards, considered sequences of r.v. that are, using the current terminology, Markov chains with a doubly stochastic transition matrix. However, Poincaré did not carry out a systematic study of these sequences. ...