Introduction to Stochastic Processes and Simulation

5Markov Processes

CONCEPTS DISCUSSED IN THIS CHAPTER.– This chapter deals with the study of Markov processes, the equivalents of Markov chains when time is considered continuous and all states remain discrete.

We will first begin by giving a precise definition of a Markov process and then study a special case: the Poisson process.

In the following section, we will look into the probability law that governs the time intervals separating two consecutive events of a Poisson process: exponential distribution. The relationship between Poisson distribution and exponential distribution will be examined.

Then we will study the general case of a birth process and a death process.

We will end with the study of the conjugation of the two preceding processes when they are in equilibrium, i.e. self-regulating, which is called the birth and death process.

The results obtained in this chapter will be used in the following chapter on queueing systems.

Recommended reading: [ENG 76, FAU 79, FOA 04, LES 14, RUE 89].

5.1. The concept of Markov processes

In Markov chains, time is considered discrete: it takes countable values. When time is considered continuous, the corresponding process is called a Markov process.

In both cases, Markov chains and processes, all the states taken by a system remain discrete: E₁, E₂,..., E_m,..., finite or infinite.

Let X(t) be a random variable taking the value i when the system is in state E_i at moment t. X(t) defines a Markov process if the following assumption ...