In this chapter, we start the study of some the most popular class of models suitable for real-life situations.
12.1 Basic Concepts for Markov Chains
Consider a set of outcomes which is finite or countable. is called the states space. It is convenient to represent the set as the nonnegative integers (any discrete set may be put into a bijection with this set). Consider a process whose components take values in this set . We will say that the process X is in state at time n if .
We next consider a matrix
with the elements
Such a matrix is often ...