MA3264 Mathematical ModellingLecture 5

Discrete Probabilistic Modelling

Markov Chains

vertices represent states and are labeled 1,..,K

Can be illustrated using labeled directed graphs

edges represent transitions between states and are labeled by numbers in the interval [0,1]

Page 217, Figure 6.1

1 2

p1

q1

qp

Transition Matrix T=

qpqp

11

NOTE Our TM = transpose book’s TM

Transition Probabilities

The label on a directed edge from vertex j to vertex i is the probability that if the system is in state j at time n then it will be in state i at time n+1 FOR EVERY n (in other words these so called transition probabilities (TP) do not change over time), for convenience we place these TP into a transition matrix T

Imagine a dynamic* system that can be in one of K states during each time interval [n,n+1)

1 2

p1

q1

qp

qpqp

11

* marked by usually continuous and productive activity or change

Transition ProbabilitiesFor the situation displayed in the graph below, if a system is in state 1 at time n = 1, then the probability that the system will change to be is state 2 at time n = 2 equals 1-p and the probability that the system will stay the same to be in state 1 at time n = 2 equals p, similar considerations apply if the system is in state 2 at time n = 1

1 2

p1

q1

qp

qpqp

11

The sum of the elements in each column of T equals 1

State VectorsIf at time n, the probability of a system being in state 1 equals a(n) and the probability of being in state 2 equals b(n) then this may be represented by a state vector

)()(

)(nbna

nv

Clearly the entries in this matrix are in [0,1] and their sum equals 1

Dynamics of State Vectors

The dynamics of this vector are derived from probability

)()()()(

)1(2221

1211

nbTnaTnbTnaT

nv

)()()(

2221

1211 nTvnbna

TTTT

Law of Large NumbersIf we have two (or more) large populations of individuals, each of whom can be in state 1 or state 2 at each time n, then a(n), b(n) can be interpreted as the frequency (or fractions) of individuals that are in state 1, 2 at time n

Likewise, if a state vector has entries that represent frequencies, then these frequencies can be interpreted as probabilities of an individual who is chosen randomly to be in state 1, 2 at time n

Likewise, the entries of a transition matrix can be either interpreted as probabilities or as frequencies

This dual interpretation aspect can be initially confusing but becomes much more obvious through applications

Rental Car ApplicationExample 1. Rental Car Company (pages 217-218)

1 24.

3.

7.6.

7.4.3.6.

T

If we let nn qp , the fraction of cars in Orlando, Tampa

at time n, then

nnn qpp 3.6.1

nnn qpq 7.4.1

n

n

n

n

qp

Tqp

1

1

11

OrlandoTampa

Long Term Behavior

Can we find the long-term behavior ?

7.4.3.6.

T

0

0

2

22

1

1

qp

Tqp

Tqp

Tqp n

n

n

n

n

n

n

This matrix notation gives n-step transitions

61.52.39.48.2T

583.556.417.444.3T

5749.5668.4251.4332.4T

5717.5710.4283.4290.6T

Long Term Behavior

If 1,lim

ba

bbaa

T k

kthen

bbaa

bbaa

T

baba

ba

T7.4.3.6.

ba

ba

T

5714.4286.

ba

MATLAB Experiment>> T = [.6 .3;.4 .7]

T =

0.6000 0.3000 0.4000 0.7000

>> for k = 1:20Tk = T^k;a1(k) = Tk(1,1);b1(k) = Tk(2,1);a2(k) = Tk(1,2);b2(k) = Tk(2,2);End

>> plot([a1' a2' b1' b2'])

Radioactive DecayModel decay of an atom of Polonium 209 to Lead 205after its half life (=102 years), its state vector evolves as

1 2

21

0

121

Transition Matrix T=

10

2121

1. Po209, 2. Pb205

1615161

8781

4341

2121

01

10

10

Radioactive DecayState vector dynamics of # Po209 atoms remaining(after 102 years)

1 2

41

1

1000

21

41

21

2141

T

1. two Po209 2. one Po209 3. zero Po209

3214

1

21

1

21

Traffic Light

With fixed transition times of one minute

1 2 1

010001100

T

1. red 2. green 3. yellow

31 1

1

001

100

010

001

100

010

001

Convergence CriteriaTheorem (Perron-Frobenius) If T is an n x n transition matrix then the following three statements are true

]111[v

ST kk

lim(ii)

(i) 1 is an eigenvalue of T since vT = v where

exists (converges) if and only if all other

(iii) If there exists an integer p > 0 such that every entry of the matrix is positive then the if condition in (ii) is satisfied and every entry of w is positive.

PT

eigenvalues have modulus < 1,then the right eigenvector w (Tw = w) for eigenvalue 1 (normalized so the sum of the entries of w equals 1), is the steady state of T and each column of S equals w

Numerical Examples

7.4.3.6.

Matrix Eigenvalues Steady State Vector

15.05.

4.2.2.2.6.05.4.2.75.

1,3.

5714.4286.

1,5.

10

1,55.,2.

2500.1944.5556.

Suggested Reading&Problems in Textbook

6.1 Probabilistic Modelling with Discrete Systems, pages 217-222

http://aix1.uottawa.ca/~jkhoury/markov.htm

Recommended Websites

http://en.wikipedia.org/wiki/Markov_chain

http://www.eng.buffalo.edu/~kofke/applets/MarkovApplet1.html

Tutorial 5 Due Week 29 Sept – 3 Oct

Page 222. Problem 1.

Page 222. Problems 2

Page 222-223. Project Problem 1