1

3.8 Fundamentals of Markov Chains

A Markov chain is a special class of state model. As with earlier state models, it consists of a

collection of states, only now we are modeling probabilities of transitions between states. The

weight assigned to each arrow is now interpreted as either the probability that something in the state

at the arrow's tail moves to the state at the arrow's head, or the percentage of things at the arrow's

tail which move to the state at the arrow's head. At each time step, something in one state must

either remain where it is or move to another state. Thus the sum of the arrows into (or out of) a state

must be one. The state vector X(i) in a Markov model traditionally lists either the probability that a

system is in a particular state at a particular time, or the percentage of the system which is in each

state at a given time. Thus X(t) is a probability distribution vector and must sum to one. We have

occasionally mentioned such vectors in what we have done before, but when dealing with a Markov

model we deal with probability distribution vectors exclusively. Recapping, there are three

properties which identify a state model as being a Markov model: 1) The Markov assumption: the

probability of one's moving from state i to state j is independent of what happened before moving to

state j and of how one got to state i. 2) Conservation: the sum of the probabilities out of a state must

be one. 3) The vector X(t) is a probability distribution vector which describes the probability of the

system's being in each of the states at time n.

In some sense, we have been assuming the Markov assumption all along. By this we mean that we

have been assuming that the number being assigned to a state during a time step depends only on

the way things were distributed during the prior time step and not any further back than that. This

was the fourth convention we made when defining state diagrams. Essentially it says that we are

considering only first-order recurrence relations. Strictly speaking the Markov assumption refers to

only probabilities, but we used equivalents of it with birth rates that were greater than one. When

discussing the probabilities associated with a Markov chain, the term conditional probability is

often used. Conditional probability means just the probability of something's happening given that

something else has already happened. In our case the probability of moving from state i to state j

assumes we were in state i to begin with, so, technically, this is a conditional probability.

The transition matrix for a Markov chain is then a matrix of probabilities (conditional probabilities

if we are perfectly correct) of moving from one state to another. Thus

We also require that the each column sums to one in order to satisfy the conservation property. The

system moves from states given by column indices to states given by row indices. For example, p21

is the probability of the system's moving from state 2 to state 1.

We can represent a Markov chain using a state diagram (Figure 3.12).

The transition probabilities pl:j are shown as the flows between states.

Stages, States, and Classes

2

FIGURE 3.12 General State diagram of a Markov Chain.

Consider the following transition matrix for a Markov chain:

There are three states for this chain, which we label i = 1,2,3. The state diagram for this chain is

shown in Figure 3.13.

Unlike models discussed earlier, the vector X(t) does not give the number of individuals in each

state at time t; rather it gives the probability that the system is in each state at time r. It is

conventional with Markov chains to denote X(t) as Xt. An initial distribution Xo, is a distribution for

the chance that the system is initially in each of the states. For instance, suppose

The interpretation X0 is that there is a 50% chance the system is initially in state 1, 30% chance it is

in state 2, and a 20% chance it is in state 3.

In this context, matrix multiplication gives the probability distribution one time step later. That is,

3

where X0 is an initial distribution. Using the transition matrix and initial distribution from above, we

have

so that after one time step, there is a 36% chance of the system's being in state 1, and 35% and 29%

chances of being in states 2 and 3 respectively. Using this notation, the distribution after t = n time

steps is given by

(62)

4

An important idea, which we make use of in the next two sections, is whether the sequence of

column vectors Xn, n > 1 converges to a steady-state (unchanging from time step to time step)

column vector . Determining allows us to answer long-term behavior questions we may

pose.

We observe here that if Xn —> as n —> oo, we must have the matrix Tn approaching some fixed

matrix L, that is

The matrix L, if it exists, is referred to as the "steady-state" matrix. The convergence of the matrix

Tn to the steady-state matrix L is independent of the initial distribution XQ, as-equation (63) shows.

The steady-state distribution and the steady-state matrix L can be shown to exist, provided that

the transition matrix T satisfies the property that some power has all positive entries. Matrices

satisfying this condition are called regular. If T is regular, we find the steady-state distribution

by solving the set of equations.

for , along with the condition that the sum of the entries in must be one. The matrix equation

(64) clearly conveys the idea that the steady-state distribution is a fixed point of the system of

equations (62). Equivalently, is an eigenvector with eigenvalue one. An intuitively appealing

method for determining the steady-state distribution , is to compute (or approximate) the steady-

state matrix L. Traditionally, this is done analytically using a method called matrix diagonalization.

Since

we approximate L by computing Tn for large values of n. This is easily done using a software

package, or, if the number of states is small, a calculator with matrix capabilities.

For our example, the steady-state matrix is approximately

The form of the matrix in equation (65) might at first glance appear surprising. If the steady-state

matrix L exists, it has the form given in (65) where each of the columns are identical. This fact

(65)

(64)

(63)

5

follows from the equation = LXo and recalling that the sum of the entries in the column vector

XQ is one. This equation also demonstrates that each column of L( ≈ Tn, for n large) is . Thus, for

our example, we have the steady-state distribution

There is another class of Markov chains which have important modeling properties. Consider the

following example of a transition matrix.

The state diagram for this transition matrix is shown in Figure 3.14.

This system has some important features. States 4 and 5 are called "absorbing" states. Once the

system enters an absorbing state, the system remains in that state from that time on. The absorbing

states are easily identified from the state diagram in that they have loops with weight one. States 4

and 5 both have loops with weight one.

Absorbing Markov chains are different in structure than those we have previously considered. An

absorbing state precludes the transition matrix from being regular. The assumption that the

transition matrix is regular is enough to ensure the existence of a steady-state matrix, but is not a

characterization. Steady-state matrices exist for absorbing.

FIGURE 3.14 State diagram for the Absorbing Markov Chain.

(66)

6

Markov chains, and the additional structure of the absorbing chains provides useful information.

The project section considers an example of a nonabsorbing, non-regular transition matrix for which

a steady-state matrix can be calculated.

If we compute the steady-state matrix for the above absorbing chain, we obtain

This matrix exhibits several properties that we need later on. Examining the structure of the

transition matrix T in equation (66), we see that it can be decomposed into blocks

of the form

The matrix I2x2 is just the 2 x 2 identity matrix, and if it was not obvious, we formed the blocks

around the identity matrix block. This decomposition is always possible for absorbing Markov

chains, though we may need to re-label the states so that the absorbing states are listed last (so the

identity matrix is in the proper position). In general, if a Markov chain has a absorbing states and b

nonabsorbing states, we can arrange the transition matrix to have the form

7

This block decomposition gives useful information about the absorbing Markov chain. The steady-

state matrix L has the form

The entries in the matrix represent the probability of being absorbed in ith

absorbing state if the system was initially in the jth nonabsorbing state. In the example,

These entries are viewed as "absorption" probabilities. For example, there is a 71.43% chance that

the system will be absorbed in state 4, given that it initially started in state 2. To understand which

state is which, refer back to the columns and rows of (67). The other entries have a similar

interpretation.

Further information is obtained from the fundamental matrix .The entries

of this matrix are the average number of times the process is in state j, given that it began in state i.

A proof of this result is in Olinick [48]. For our example, the fundamental matrix is

Recalling the block form of the transition matrix (68), the position of the submatrix A indicates that

i and j have values 1, 2, or 3, so that f1,1 = 1.25 is the average number of time steps that the system

is in state 1, given that it was initially in state 1. The other entries have analogous interpretations.

The sum of the entries of the jth column of the fundamental matrix F is the average number of time

steps for a process initially in state j to be absorbed. For example, if the system is initially in state 1,

it takes an average of 1.25 + 0.7143 + 0.7143 = 2.6786 time steps before the system enters an

absorbing state. The next two sections present models based upon Markov chains and use the above

analysis. The project section also contains some interesting Markov models, as well as some further

points of the theory of Markov chains.

3.9 Markovian Squirrels

The American gray squirrel (Sciurus carolinensis Gmelin) was introduced in Great Britain by a

series of releases from various sites starting in the late nineteenth century. In 1876, the first gray

8

squirrels were imported from North America, and have subsequently spread throughout England

and Wales, as well as parts of Scotland and Ireland.

Simultaneously, the native red squirrel (Sciurus vulgaris L.), considered the endemic subspecies,

has disappeared from most of the areas colonized by gray squirrels. Originally, the red squirrel was

distributed throughout Europe and eastward to northern China, Korea, and parts of the Japanese

archipelago. During the last century, the red squirrel has consistently declined, becoming extinct in

many areas of England and Wales, so that it is now confined almost solely to Northern England and

Scotland. A few isolated red squirrel populations exist on offshore islands in southern England and

mountainous Wales.

The introduction of the American gray squirrel continued until the early 1920s, by which time the

gray squirrels had rapidly spread throughout England. By 1930 it was apparent that the gray squirrel

was a pest in deciduous forests, and control measures were attempted. Once the pest status of the

gray squirrel was recognized, national distribution surveys were undertaken. The resulting

distribution maps clearly showed the tendency for the red squirrel to be lost from areas that had

been colonized by the gray squir during the preceding 15 to 20 years.

Since 1973, an annual questionnaire has been circulated to foresters by the British Forestry

Commission. The questionnaire concerns the presence or absence of the m squirrel species. It also

includes questions on the changes of squirrel abundance, details of tree damage, squirrel control

measures, and the number of squirrels killed. Using c data collected by the Forestry Commission,

we wish to construct a model to predict the trends in the distribution of both species of squirrels in

Great Britain.

Several researchers have studied the British squirrel populations, notably Reynolds [53]. and Usher

et al. [68]. The annual Forestry Commission data has been summarized in tl form of distribution

maps reflecting change over a two-year period.

Usher et al., [68] used an overlay technique to extract data from the distribution map Each 10-km

square on the overlay map that contained Forestry Commission land classified into one of four

states:

R: only red squirrels recorded in that year.

G: only gray squirrels recorded in that year.

B: both species of squirrels recorded in that year.

O: neither species of squirrels recorded in that year.

In order to satisfy the Markov assumption, squares that were present only in tv.: consecutive years

were counted. Counting the pairs of years, squares are allocated to any one of 16 classes, e.g., R ->

R, R -> G, G -> G, B -> O, etc.

A summary of these transition counts for each pair of years from 1973-74 to 1987-88 is given in

table 3.3 and is reprinted by permission of Blackwell Science Inc.

A frequency interpretation is required to employ the Markov chain analysis. If the entries in each

column are totaled, the corresponding matrix entry is found by division. For example, column R has

a total 2.529 + 61 + 282 + 3 = 2,875, so that the entry in the R, R position is 2,529/2,875 0.8797.

Care must be taken when calculating these frequencies. Inappropriate rounding will violate the

requirement that the columns sum to

9

TABLE 3.3 Red and Gray Squirrel Distribution Map Data for Great Britain.

3 Stages, States, and Classes

FIGURE 3.15 State diagram for the Markov Squirrels.

one. The transition matrix (rows and columns are in R, G, B, O order) is

The state diagram of this transition matrix T is given in Figure 3.15. We interpret these transition

frequencies as conditional probabilities. For example, there is an 87.97% chance that squares that

10

are currently in state R (red squirrels only) will remain in state R; similarly, there is a 2.73% chance

of squares that are currently occupied by both squirrel species, state G, will become occupied by

neither species, state B, after the next time step. Since the data taken from the annual Forestry

Commission survey is summarized as pairs of years, each time step represents a two-year period.

The matrix form of the transition probabilities is convenient for calculations. Using matrix

multiplication, we compute the two-time-step transition matrix as T2 = T x T, which is given by

The entries of this transition matrix are again interpreted as conditional probabilities. For instance,

there is a 17.33% chance that squares currently occupied by only red squirrels, state R, will be

occupied by both species, state B, in two time steps (four years).

Using the transition matrix T, it is possible to gain insight into the long-term behavior of the two

species of squirrels. We compute the steady-state matrix L for the two squirrel populations. The

question of interest in the study of the squirrel populations is what happens to the distribution of the

squirrel populations over a long period of time.

For our squirrel model, the steady-state matrix is approximately

Thus the steady-state distribution is

This result is interpreted as the long-term behavior of the squirrel populations in Great Britain as

follows: 17.05% of the squares will be in state R, containing only red squirrels. There will be 5.6%

of the squares in state G containing only gray squirrels. There will be populations of both squirrels

11

in 34.21% of the squares (state B), with the majority of the squares, 43.14%, being occupied by

neither species of squirrels (state O).

If the assumptions made in this model are correct, the red squirrel is not currently in danger. In fact,

it will have sole possession of more regions than the gray squirrel will have. In the long term, the

gray squirrels do not drive the reds to extinction. Actually this analysis says nothing about

population sizes, only about the number of regions controlled by each type of squirrel. While it

seems plausible that if the red squirrel territory (number of regions) is declining, then the population

is declining; the opposite may be true. A problem in the projects section asks you to perform this

analysis for the two squirrel species in Scotland, where the red squirrel is still widely distributed.