Markov Chain - Pennsylvania State Universitypersonal.psu.edu/jol2/course/stat416/notes/chap4.pdf ·...
Transcript of Markov Chain - Pennsylvania State Universitypersonal.psu.edu/jol2/course/stat416/notes/chap4.pdf ·...
Markov Chain
• Stochastic process (discrete time):
{X1, X2, ..., }
• Markov chain
– Consider a discrete time stochastic process withdiscrete space.Xn ∈ {0, 1, 2, ...}.
– Markovian property
P{Xn+1 = j | Xn = i,Xn−1 = in−1, ..., X0 = i0}= P{Xn+1 = j | Xn = i} = Pi,j
– Pi,j is thetransition probability: the probability ofmaking a transition fromi to j.
– Transition probability matrix
P =
P0,0 P0,1 P0,2 · · ·P1,0 P1,1 P1,2 · · ·
... ... ... ...Pi,0 Pi,1 Pi,2 · · ·
... ... ... ...
1
Example
• Suppose whether it will rain tomorrow depends onpast weather condition ony through whether it rainstoday. Consider the stochastic process{Xn, n = 1, 2, ...}
Xn =
{
0 rain on dayn1 not rain on dayn
P (Xn+1|Xn, Xn−1, ..., X1) = P (Xn+1 | Xn)
• State space{0, 1}.
• Transition matrix:(
P0,0 P0,1
P1,0 P1,1
)
• P0,0 = P (tomorrow rain|today rain) = α. ThenP0,1 =1 − α.
• P1,0 = P (tomorrow rain|today not rain) = β. ThenP1,1 = 1 − β.
• Transition matrix:(
α 1 − αβ 1 − β
)
2
Transforming into a Markov Chain
• Suppose whether it will rain tomorrow depends onwhether it rained today and yesterday.
• P (Xn+1|Xn, Xn−1, ..., X1) = P (Xn+1|Xn, Xn−1). Theprocess is not a first order Markov chain.
• DefineYn:
Yn =
0 Xn = 0, Xn−1 = 0 RR1 Xn = 0, Xn−1 = 1 NR2 Xn = 1, Xn−1 = 0 RN3 Xn = 1, Xn−1 = 1 NN
•P (Yn+1|Yn, Yn−1, ...)
= P (Xn+1, Xn|Xn, Xn−1, ...)
= P (Xn+1, Xn|Xn, Xn−1)
= P (Yn+1|Yn)
• {Yn, n = 1, 2, ...} is a Markov chain.
P0,0 P0,1 P0,2 P0,3
P1,0 P1,1 P1,2 P1,3
P2,0 P2,1 P2,2 P2,3
P3,0 P3,1 P3,2 P3,3
3
• P0,1 = P (Yn+1 = 1|Yn = 0) = P (Xn+1 = 0, Xn =1|Xn = 0, Xn−1 = 0) = 0.
• P0,3 = P (Yn+1 = 3|Yn = 0) = P (Xn+1 = 1, Xn =1|Xn = 0, Xn−1 = 0) = 0.
• Similarly, P1,1 = P1,3 = 0, P2,0 = P2,2 = 0, P3,0 =P3,2 = 0.
• Transition matrix
P0,0 0 P0,2 0P1,0 0 P1,2 00 P2,1 0 P2,3
0 P3,1 0 P3,3
=
P0,0 0 1 − P0,0 0P1,0 0 1 − P1,0 00 P2,1 0 1 − P2,1
0 P3,1 0 1 − P3,1
• The Markov chain is specified byP0,0, P1,0, P2,1, P3,1.
1. P0,0 = P (tomorrow will rain|today rain, yesterday rain).
2. P1,0 = P (tomorrow will rain|today rain, yesterday not rain).
3. P2,1 = P (tomorrow will rain|today not rain, yesterday rain).
4. P3,1 = P (tomorrow will rain|today not rain, yesterday not rain).
4
Chapman-Kolmogorov Equations
• Transition afternth steps:P n
i,j = P (Xn+m = j | Xm = i).
• Chapman-Kolmogorov Equations:
P n+mi,j =
∞∑
k=0
P ni,kP
mk,j, n, m ≥ 0 for all i, j.
• Proof (by Total probability formula):
P n+mi,j = P (Xn+m = j|X0 = i)
=∞
∑
k=0
P (Xn+m = j,Xn = k|X0 = i)
=∞
∑
k=0
P (Xn = k|X0 = i) ·
P (Xn+m = j|Xn = k,X0 = i)
=∞
∑
k=0
P ni,kP
mk,j
5
• n-step transition matrix:
P(n) =
P n0,0 P n
0,1 P n0,2 · · ·
P n1,0 P n
1,1 P n1,2 · · ·
... ... ... ...P n
i,0 P ni,1 P n
i,2 · · ·... ... ... ...
• Chapman-Kolmogorov Equations:
P(n+m) = P
(n) · P(m), P(n) = P
n.
• Weather example:
P =
(
α 1 − αβ 1 − β
)
=
(
0.7 0.30.4 0.6
)
FindP (rain on Tuesday| rain on Sunday) andP (rain on Tuesday and rain on Wednesday| rain on Sunday).
Solution:
P (rain on Tuesday| rain on Sunday) = P 20,0
P(2) = P · P =
(
0.7 0.30.4 0.6
)
×(
0.7 0.30.4 0.6
)
=
(
0.61 0.390.52 0.48
)
P (rain on Tuesday| rain on Sunday) = 0.61
6
P (rain on Tuesday and rain on Wednesday| rain on Sunday)
= P (Xn = 0, Xn+1 = 0 | Xn−2 = 0)
= P (Xn = 0|Xn−2 = 0)P (Xn+1 = 0|Xn = 0, Xn−2 = 0)
= P (Xn = 0|Xn−2 = 0)P (Xn+1 = 0|Xn = 0)
= P 20,0P0,0
= 0.61 × 0.7 = 0.427
7
Classification of States
• Accessible: Statej is accessible from statei if P ni,j >
0 for somen ≥ 0.
– i → j.
– Equivalent to:P (ever enterj|start ini) > 0.
P (ever enterj|start ini)
= P (∪∞n=0{Xn = j}|X0 = i)
≤∞
∑
n=0
P (Xn = j | X0 = i)
=∞
∑
n=0
P ni,j
Hence ifP ni,j = 0 for all n, P (ever enterj|start ini) =
0. On the other hand,
P (ever enterj|start ini)
= P (∪∞n=0{Xn = j}|X0 = i)
≥ P ({Xn = j}|X0 = i) for anyn
= P ni,j .
If P ni,j > 0 for somen, P (ever enterj|start ini) ≥
P ni,j > 0.
– Examples
8
• Communicate: Statei andj communicate if they areaccessible from each other.
– i ↔ j.
– Properties:
1. P 0i,i = P (X0 = i|X0 = i) = 1. Any statei
communicates with itself.2. If i ↔ j, thenj ↔ i.3. If i ↔ j andj ↔ k, theni ↔ k.
Proof:
i ↔ j =⇒ P ni,j > 0 andP n′
j,i > 0
j ↔ k =⇒ Pmj,k > 0 andPm′
k,j > 0
P n+mi,k =
∞∑
l=0
P ni,lP
ml,k Chapman-Kolmogorov Eq.
> P ni,j · Pm
j,k
> 0
Similarly, we can showP n′+m′k,i > 0. Hencei ↔
k.
9
• Class: Two states that communciate are said to bein the same class. A class is a subset of states thatcommunicate with each other.
– Different classes do NOT overlap.
– Classes form a partition of states.
• Irreducible: A Markov chain is irreducible if there isonly one class.
– Consider the Markov chain with transition proba-bility matrix:
P =
12
12 0
12
14
14
0 13
23
The MC is irreducible.
– MC with transition probability matrix:
P =
12
12
0 012
12 0 0
14
14
14
14
0 0 0 1
Three classes:{0, 1}, {2}, {3}.
10
Recurrent and Transient States
• fi: probability that starting in statei, the MC will everreenter statei.
• Recurrent: If fi = 1, statei is recurrent.
– A recurrent states will be visited infinitely manytimes by the process starting fromi.
• Transient: If fi < 1, statei is transient.
– Starting fromi, the MC will be in statei for ex-actlyn times (including the starting state) is
fn−1i (1 − fi) , n = 1, 2, ...
This is a geometric distribution with parameter1−fi. The expected number of times spent in stateiis 1/(1 − fi).
11
• A state is recurrentif and only if the expected numberof time periods that the process is in statei, startingfrom statei, is infinite.
Recurrent⇐⇒ E(number of visits toi|X0 = i) = ∞Transient⇐⇒ E(number of visits toi|X0 = i) < ∞
• ComputeE(number of visits toi|X0 = i). Define
In =
{
1 if Xn = i0 if Xn 6= i
Then the number of visits toi is∑∞
n=0 In.
E(number of visits toi|X0 = i)
=∞
∑
n=0
E(In|X0 = i)
=
∞∑
n=0
P (In = 1|X0 = i)
=∞
∑
n=0
P (Xn = i|X0 = i)
=∞
∑
n=0
P ni,i
12
• Proposition 4.1: Statei is recurrent if∑∞
n=0 P ni,i =
∞, and transient if∑∞
n=0 P ni,i < ∞.
• Corollary 4.2: If statei is recurrent and statei com-municates with statej, then statej is recurrent.
• Corollary 4.3: A finite state Markov chain cannothave all transient states.
– For any irreducible and finite-state Markov chain,all states are recurrent.
13
• Consider a MC with
P =
0 0 12
12
1 0 0 00 1 0 00 1 0 0
The MC is irreducible and finite state, hence all statesare recurrent.
• Consider a MC with
P =
12
12 0 0 0
12
12
0 0 00 0 1
212 0
0 0 12
12
014
14 0 0 1
2
Three classes:{0, 1}, {2, 3}, {4}. State0, 1, 2, 3 arerecurrent and state4 is transient.
14
Random Walk
• A Markov chain with state spacei = 0,±1,±2, ....
• Transition probability:Pi,i+1 = p = 1 − Pi,i−1.
– At every step, move either 1 step forward or 1 stepbackward.
• Example: a gambler either wins a dollar or loses adollar at every game.Xn is the number of dollars hehas when starting thenth game.
• For anyi < j, P j−ii,j = pj−i > 0, P j−i
j,i = (1 − p)j−i >0. The MC is irreducible.
• Hence, either all the states are transient or all thestates are recurrent.
15
• Under which condition are the states transient or re-current?
– Consider State0.∞
∑
n=1
P n0,0 =
{
∞ recurrentfinite transient
– Only for evenm, Pm0,0 > 0.
P 2n0,0 =
(
2nn
)
pn(1 − p)n =(2n)!
n!n!(p(1 − p))n
n = 1, 2, 3, ...
– By Stirling’s approximation
n! ∼ nn+1/2e−n√
2π
– P 2n0,0 ∼ (4p(1−p))n√
πn.
– Whenp = 1/2, 4p(1 − p) = 1.∞
∑
n=0
(4p(1 − p))n√πn
=∞
∑
n=0
1√πn
,
The summation diverges. Hence, all the states arerecurrent.
– Whenp 6= 1/2, 4p(1 − p) < 1.∑∞
n=0(4p(1−p))n√
πn
converges. All the states are transient.
16
Limiting Probabilities
• Weather example
P =
(
0.7 0.30.4 0.6
)
•P
(4) = P4 =
(
0.5749 0.42510.5668 0.4332
)
P(8) = P
(4)P
(4) =
(
0.572 0.4280.570 0.430
)
• P(4) andP
(8) are close. The rows inP(8) are close.
• Limiting probabilities?
17
• Period d: For statei, if P ni,i = 0 whenevern is not
divisible by d and d is the largest integer with thisproperty,d is the period of statei.
– Periodd is the greatest common divisor of all them such thatPm
i,i > 0.
• Aperiodic: Statei is aperiodic if its period is1.
• Positive recurrent: If a statei is recurrent and the ex-pected time until the process returns to statei is finite.
– If i ↔ j andi is positive recurrent, thenj is posi-tive recurrent.
– For a finite-state MC, a recurrent state is also pos-itive recurrent.
– A finite-state irreducible MC contains all positiverecurrent states.
• Ergodic: A positive recurrent and aperiodic state isan ergodic state.
• A Markov chain is ergodic if all its states are ergodic.
18
• Theorem 4.1: For an irreducible ergodic Markov chain,limn→∞ P n
i,j exists and is independent ofi. Let πj =limn→∞ P n
i,j, j ≥ 0, thenπj is the unique nonnegativesolution of
πj =∞
∑
i=0
πiPi,j j ≥ 0
∞∑
j=0
πj = 1 .
• The Weather Example:
P =
(
α 1 − αβ 1 − β
)
The MC is irreducible and ergodic.
π0 + π1 = 1
π0 = π0P0,0 + π1P1,0 = π0α + π1β
π1 = π0P0,1 + π1P1,1 = π0(1 − α) + π1(1 − β)
Solve the linear equations,π0 = β1+β−α
, π1 = 1−α1+β−α
.
19
Gambler’s Ruin Problem
• At each play, a gambler either wins a unit with prob-ability p or loses a unite with probabilityq = 1 − p.Suppose the gambler starts withi units, what is theprobability that the gambler’s fortune will reachNbefore reaching0 (broke)?
• Solution:
– Let Xt be the number of units the gambler has attime t. {Xt; t = 0, 1, 2, ...} is a Markov chain.
– Transition probabilities:
P0,0 = PN,N = 1
Pi,i+1 = p, Pi,i−1 = 1 − p = q .
– Denote the probability that starting fromi units,the gambler’s fortune will reachN before reaching0 by Pi, i = 0, 1, ..., N .
– Condition on the result of the first game and applythe total probability formula:
Pi = pPi+1 + qPi−1 .
20
– Changing forms:
(p + q)Pi = pPi+1 + qPi−1
q(Pi − Pi−1) = p(Pi+1 − Pi)
Pi+1 − Pi =q
p(Pi − Pi−1)
– Recursion:
P0 = 0
P2 − P1 =q
p(P1 − P0) =
q
pP1
P3 − P2 =q
p(P2 − P1) =
(
q
p
)2
P1
...
Pi − Pi−1 =q
p(Pi−1 − Pi−2) =
(
q
p
)i−1
P1
– Add up the equations:
Pi = P1[1 +q
p+ · · · + (
q
p)i−1]
=
P1 ·1−(q
p)i
1−qp
if qp 6= 1
iP1 if qp = 1
– We knowPN = 1
PN =
P1 ·1−(q
p)N
1−qp
if qp 6= 1
NP1 if qp
= 121
– Hence,
P1 =
{
1−qp
1−(qp)N
if p 6= 1/2
1/N if p = 1/2
– In summary,
Pi =
1−(qp)i
1−(qp)N
if p 6= 1/2
i/N if p = 1/2
• Note, if N → ∞,
Pi =
{
1 − (qp)
i if p > 1/2
0 if p ≤ 1/2
• Whenp > 1/2, there is a positive probability that thegambler will win infinitely many units.
• When p ≤ 1/2, the gambler will surely go broke(with probability 1) if not stop at a finite fortune (as-suming the opponent is infinitely rich).
22