Probability and Stochastic Processes

Chapter 13Semi-Markov Processes and Continuous time Markov Processes

In many situations, we wish to model the time between transitions as well as the nature of the transitions between states. Often, we have a pretty good idea about the distribution of the time spent in a particular state. The semi-Markov process is an example of a process that tries to capture both time and state transitions. However, this process is not Markov in general, in the sense that next transitions may depend on the previous states besides the current one. In a special case where the transitions only depend on the current states, we obtain the Markov process as a particular case of the process presented in this chapter. We decided to include the semi-Markov process in this book since the mathematical treatment of the two processes is very similar. Naturally, since the Markov process is a particular case, it will have extra properties, but the mathematical framework is identical.

Definition 13.1

Define the state space as . Suppose that we have a process which jumps between these states according to a Markov chain with transition probability matrix .

However, unlike in the discrete-time Markov chain case of the previous chapter, we may stay in the state i for a random amount of time which may be dependent ...