A First Course in Probability and Markov Chains

6

An introduction to continuous time Markov chains

In this chapter, we give a very short introduction to continuous time Markov chains. In Section 6.1 we discuss the Poisson process while in Section 6.2 we consider homogeneous continuous time Markov chains with finite state-space and right-continuous trajectories. The convergence to equilibrium of the transition probabilities matrices and the description of the holding times are discussed.

6.1   Poisson process

We go back to Example 3.34 by giving an explicit example: the emission of clicks of a Geiger counter. In this section we follow closely the presentation in [5].

Evidence shows the following features of the distribution of the number of clicks produced by the counter:

(i) The number of clicks produced in pairwise disjoint time intervals are independent.

(ii) The clicks are uniformly distributed with respect to time, i.e. the number of clicks produced in the intervals [a + c, b + c] a, b, c ≥ 0 does not depend on c.

(iii) In each time interval, the average number of clicks is finite,

(iv) At each moment, the Geiger counter produces at most one click.

When defining a model, the previous features can be formalized as follows. Let be the family of time intervals

Consider the Geiger counter as a probability space (Ω, ε, ) and, ...