Chapter 12Markov Chains

In this chapter, we start the study of some the most popular class of models suitable for real-life situations.

12.1 Basic Concepts for Markov Chains

12.1.1 Definition

Consider a set of outcomes c12-math-0001 which is finite or countable. c12-math-0002 is called the states space. It is convenient to represent the set c12-math-0003 as the nonnegative integers c12-math-0004 (any discrete set may be put into a bijection with this set). Consider a process c12-math-0005 whose components c12-math-0006 take values in this set c12-math-0007. We will say that the process X is in state c12-math-0009 at time n if c12-math-0011.

We next consider a matrix

with the elements

Such a matrix is often ...

Get Probability and Stochastic Processes now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.