Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative...
Transcript of Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative...
![Page 1: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/1.jpg)
Quantitative VerificationChapter 3: Markov chains
Jan Kretınsky
Technical University of Munich
Winter 2017/18
1 / 84
![Page 2: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/2.jpg)
Motivation
2 / 84
![Page 3: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/3.jpg)
Example: Simulation of a die by coins
Knuth & Yao dieSimulating a Fair Die by a Fair Coin
1 2 3 4 5 6
s!1,2,3
s2,3 s4,5 s!4,5,6
s1,2,3 s4,5,6
s0
1 1 1 1 1 1
12
12
12
12
12
12
12
12
12
12
12 1
2
12
12
Quiz
Is the probability of obtaining 3 equal to 16?
Zhang (Saarland University, Germany) Quantitative Model Checking August 24th , 2009 12 / 1
Question:I What is the probability of obtaining 2?
3 / 84
![Page 4: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/4.jpg)
DTMC - Graph-based Definition
Definition:A discrete-time Markov chain (DTMC) is a tuple (S ,P, π0) where
I S is the set of states,I P : S × S → [0, 1] with
∑s′∈S P(s, s ′) = 1 is the transitions
matrix, andI π0 ∈ [0, 1]|S| with
∑s∈S π0(s) = 1 is the initial distribution.
4 / 84
![Page 5: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/5.jpg)
Example: Craps
Two dice game:I First:
∑∈7, 11 ⇒ win,∑∈2, 3, 12 ⇒ lose, else s=
∑I Next rolls:
∑= s ⇒ win,
∑= 7⇒ lose, else iterate
Modelling Markov Chains
init
won
lost
29
19
Zhang (Saarland University, Germany) Quantitative Model Checking August 24th , 2009 14 / 35
5 / 84
![Page 6: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/6.jpg)
Example: Craps
Two dice game:I First:
∑∈7, 11 ⇒ win,∑∈2, 3, 12 ⇒ lose, else s=
∑I Next rolls:
∑= s ⇒ win,
∑= 7⇒ lose, else iterate
Modelling Markov Chains
init
won
1
lost
1
29
19
4 5 6 8 9 10
112
19
536
536
19
112
Zhang (Saarland University, Germany) Quantitative Model Checking August 24th , 2009 15 / 35
6 / 84
![Page 7: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/7.jpg)
Example: Craps
Two dice game:I First:
∑∈7, 11 ⇒ win,∑∈2, 3, 12 ⇒ lose, else s=
∑I Next rolls:
∑= s ⇒ win,
∑= 7⇒ lose, else iterate
Modelling Markov Chains
init
won
1
lost
1
29
19
4 5 6 8 9 10
112
112
16
34
19
19
16
1318
536
536
16
2536
536
536
16
2536
19
19
16
1318
112
112
16
34
Zhang (Saarland University, Germany) Quantitative Model Checking August 24th , 2009 16 / 35
7 / 84
![Page 8: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/8.jpg)
Example: Zero Configuration Networking (Zeroconf)I Previously: Manual assignment of IP addressesI Zeroconf: Dynamic configuration of local IPv4 addressesI Advantage: Simple devices able to communicate automatically
Automatic Private IP Addressing (APIPA) – RFC 3927I Used when DHCP is configured but unavailableI Pick randomly an address from 169.254.1.0 – 169.254.254.255I Find out whether anybody else uses this address (by sending
several ARP requests)
Model:I Randomly pick an address among the K (65024) addresses.I With m hosts in the network, collision probability is q = m
K .I Send 4 ARP requests.I In case of collision, the probability of no answer to the ARP
request is p (due to the lossy channel)8 / 84
![Page 9: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/9.jpg)
Example: Zero Configuration Networking (Zeroconf)
Zeroconf as a Markov chain
s0 s1 s2 s3 s4
s5
s6
s7
s8
q p p p
p
1
1 ! q
1
1 ! p1 ! p
1 ! p
1 ! p
start
error
Zhang (Saarland University, Germany) Quantitative Model Checking August 24th , 2009 19 / 35
For 100 hosts and p = 0.001, the probability of error is ≈ 1.55 · 10−15.
9 / 84
![Page 10: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/10.jpg)
Probabilistic Model Checking
What is probabilistic model checking?I Probabilistic specifications, e.g. probability of reaching bad
states shall be smaller than 0.01.I Probabilistic model checking is an automatic verification
technique for this purpose.
Why quantities?I Randomized algorithmsI Faults e.g. due to the environment, lossy channelsI Performance analysis, e.g. reliability, availability
10 / 84
![Page 11: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/11.jpg)
Basics of Probability Theory(Recap)
11 / 84
![Page 12: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/12.jpg)
What are probabilities? - Intuition
Throwing a fair coin:I The outcome head has a probability of 0.5.I The outcome tail has a probability of 0.5.
But . . . [Bertrand’s Paradox]Draw a random chord on the unit circle. What is the probability thatits length exceeds the length of a side of the equilateral triangle inthe circle?
13
12
14
12 / 84
![Page 13: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/13.jpg)
What are probabilities? - Intuition
Throwing a fair coin:I The outcome head has a probability of 0.5.I The outcome tail has a probability of 0.5.
But . . . [Bertrand’s Paradox]Draw a random chord on the unit circle. What is the probability thatits length exceeds the length of a side of the equilateral triangle inthe circle?
13
12
14
12 / 84
![Page 14: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/14.jpg)
What are probabilities? - Intuition
Throwing a fair coin:I The outcome head has a probability of 0.5.I The outcome tail has a probability of 0.5.
But . . . [Bertrand’s Paradox]Draw a random chord on the unit circle. What is the probability thatits length exceeds the length of a side of the equilateral triangle inthe circle?
13
12
14
12 / 84
![Page 15: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/15.jpg)
What are probabilities? - Intuition
Throwing a fair coin:I The outcome head has a probability of 0.5.I The outcome tail has a probability of 0.5.
But . . . [Bertrand’s Paradox]Draw a random chord on the unit circle. What is the probability thatits length exceeds the length of a side of the equilateral triangle inthe circle?
13
12
14
12 / 84
![Page 16: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/16.jpg)
Probability Theory - Probability Space
Definition: Probability FunctionGiven sample space Ω and σ-algebra F , a probability functionP : F → [0, 1] satisfies:
I P(A) ≥ 0 for A ∈ F ,I P(Ω) = 1, andI P(
⋃∞i=1Ai ) =
∑∞i=1 P(Ai ) for pairwise disjoint Ai ∈ F
Definition: Probability SpaceA probability space is a tuple (Ω,F ,P) with a sample space Ω,σ-algebra F ⊆ 2Ω and probability function P .
ExampleA random real number taken uniformly from the interval [0, 1].
I Sample space: Ω = [0, 1].
I σ-algebra: F is the minimal superset of [a, b] | 0 ≤ a ≤ b ≤ 1closed under complementation and countable union.
I Probability function: P([a, b]) = (b − a), by Caratheodory’s extensiontheorem there is a unique way how to extend it to all elements of F .
13 / 84
![Page 17: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/17.jpg)
Probability Theory - Probability Space
Definition: Probability FunctionGiven sample space Ω and σ-algebra F , a probability functionP : F → [0, 1] satisfies:
I P(A) ≥ 0 for A ∈ F ,I P(Ω) = 1, andI P(
⋃∞i=1Ai ) =
∑∞i=1 P(Ai ) for pairwise disjoint Ai ∈ F
Definition: Probability SpaceA probability space is a tuple (Ω,F ,P) with a sample space Ω,σ-algebra F ⊆ 2Ω and probability function P .
ExampleA random real number taken uniformly from the interval [0, 1].
I Sample space: Ω = [0, 1].I σ-algebra: F is the minimal superset of [a, b] | 0 ≤ a ≤ b ≤ 1
closed under complementation and countable union.I Probability function: P([a, b]) = (b − a), by Caratheodory’s extension
theorem there is a unique way how to extend it to all elements of F .13 / 84
![Page 18: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/18.jpg)
Random Variables
14 / 84
![Page 19: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/19.jpg)
Random Variables - Introduction
Definition: Random VariableA random variable X is a measurable function X : Ω→ I to some I .Elements of I are called random elements. Often I = R:
0
!
"
X
42
Example (Bernoulli Trials)Throwing a coin 3 times: Ω3 = hhh, hht, hth, htt, thh, tht, tth, ttt.We define 3 random variables Xi : Ω→ h, t. For all x , y , z ∈ h, t,
I X1(xyz) = x ,I X2(xyz) = y ,I X3(xyz) = z .
15 / 84
![Page 20: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/20.jpg)
Stochastic Processes andMarkov Chains
16 / 84
![Page 21: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/21.jpg)
Stochastic Processes - Definition
Definition:Given a probability space (Ω,F ,P), a stochastic process is a familyof random variables
Xt | t ∈ Tdefined on (Ω,F ,P). For each Xt we assume
Xt : Ω→ S
where S = s1, s2, . . . is a finite or countable set called state space.
A stochastic process Xt | t ∈ T is calledI discrete-time if T = N orI continuous-time if T = R≥0.
For the following lectures we focus on discrete time.
17 / 84
![Page 22: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/22.jpg)
Discrete-time Stochastic Processes - Construction (1)
Example: Weather ForecastI S = sun, rain,I we model time as discrete – a random variable for each day:
I X0 is the weather today,I Xi is the weather in i days.
I how can we set up the probability space to measure e.g.P(Xi = sun)?
18 / 84
![Page 23: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/23.jpg)
Discrete-time Stochastic Processes - Construction (2)
Let us fix a state space S . How can we construct the probabilityspace (Ω,F ,P)?
Definition: Sample Space ΩWe define Ω = S∞. Then, each Xn maps a sample ω = ω0ω1 . . . ontothe respective state at time n, i.e.,
(Xn)(ω) = ωn ∈ S .
19 / 84
![Page 24: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/24.jpg)
Discrete-time Stochastic Processes - Construction (3)
Definition: Cylinder SetFor s0 · · · sn ∈ Sn+1, we set the cylinder C (s0 . . . sn) = s0 · · · sn ω ∈ Ω.
Example:S = s1, s2, s3 and C (s1s3)
s1
s2
s3
s1
s2
s3
s1
s2
s3
s1
s2
s3
s1
s2
s3
s1
s2
s3
...
...
...t
S
Definition: σ-algebra FWe define F to be the smallest σ-Algebra that contains all cylindersets, i.e.,
C (s0 . . . sn) | n ∈ N, si ∈ S ⊆ F .
Check: Is each Xi measurable?(on the discrete set S we assume the full σ-algebra 2S ).
20 / 84
![Page 25: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/25.jpg)
Discrete-time Stochastic Processes - Construction (4)
How to specify the probability Function P?We only needs to specify for each s0 · · · sn ∈ Sn
P(C (s0 . . . sn)).
This amounts to specifying1. P(C (s0)) for each s0 ∈ S , and2. P(C (s0 . . . si ) | C (s0 . . . si−1)) for each s0 · · · si ∈ S i
since
P(C (s0 . . . sn)) = P(C (s0 . . . sn) | C (s0 . . . sn−1)) · P(C (s0 . . . sn−1))
= P(C (s0)) ·n∏
i=1
P(C (s0 . . . si ) | C (s0 . . . si−1))
Still, lots of possibilities...
21 / 84
![Page 26: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/26.jpg)
Discrete-time Stochastic Processes - Construction (4)
How to specify the probability Function P?We only needs to specify for each s0 · · · sn ∈ Sn
P(C (s0 . . . sn)).
This amounts to specifying1. P(C (s0)) for each s0 ∈ S , and2. P(C (s0 . . . si ) | C (s0 . . . si−1)) for each s0 · · · si ∈ S i
since
P(C (s0 . . . sn)) = P(C (s0 . . . sn) | C (s0 . . . sn−1)) · P(C (s0 . . . sn−1))
= P(C (s0)) ·n∏
i=1
P(C (s0 . . . si ) | C (s0 . . . si−1))
Still, lots of possibilities...21 / 84
![Page 27: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/27.jpg)
Discrete-time Stochastic Processes - Construction (5)
Weather Example: Option 1 - statistics of days of a yearI the forecast starts on Jan 01,I a distribution pj over sun, rain for
each 1 ≤ j ≤ 365,I for each i ∈ N and s0 · · · si ∈ S i+1
P(C (s0 . . . si ) | C (s0 . . . si−1)) = pi % 365(si )
Weather Example: Option 2 - two past daysI a distribution ps′s′′ over sun, rain
for each s ′, s ′′ ∈ S ,I for each i ≥ 2 and s0 · · · si ∈ S i+1
P(C (s0 . . . si ) | C (s0 . . . si−1)) = psi−2si−1 (si )
Here: time-homogeneous Markovian stochastic processes
Not Markovian.
Not time-homogeneous.
22 / 84
![Page 28: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/28.jpg)
Discrete-time Stochastic Processes - Construction (5)
Weather Example: Option 1 - statistics of days of a yearI the forecast starts on Jan 01,I a distribution pj over sun, rain for
each 1 ≤ j ≤ 365,I for each i ∈ N and s0 · · · si ∈ S i+1
P(C (s0 . . . si ) | C (s0 . . . si−1)) = pi % 365(si )
Weather Example: Option 2 - two past daysI a distribution ps′s′′ over sun, rain
for each s ′, s ′′ ∈ S ,I for each i ≥ 2 and s0 · · · si ∈ S i+1
P(C (s0 . . . si ) | C (s0 . . . si−1)) = psi−2si−1 (si )
Here: time-homogeneous Markovian stochastic processes
Not Markovian.
Not time-homogeneous.
22 / 84
![Page 29: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/29.jpg)
Stochastic Processes - Restrictions
Definition: MarkovA discrete-time stochastic processXn | n ∈ N is Markov if
P(Xn = sn | Xn−1 = sn−1, . . . ,X0 = s0)
= P(Xn = sn | Xn−1 = sn−1)
for all n > 1 and s0, . . . , sn ∈ S withP(Xn−1 = sn−1) > 0.
Definition: Time-homogeneousA discrete-time Markov process Xn | n ∈ Nis time-homogeneous if
P(Xn+1 = s ′ | Xn = s) = P(X1 = s ′ | X0 = s)
for all n > 1 and s, s ′ ∈ S with P(X0 = s) > 0.
23 / 84
![Page 30: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/30.jpg)
Stochastic Processes - Restrictions
Definition: MarkovA discrete-time stochastic processXn | n ∈ N is Markov if
P(Xn = sn | Xn−1 = sn−1, . . . ,X0 = s0)
= P(Xn = sn | Xn−1 = sn−1)
for all n > 1 and s0, . . . , sn ∈ S withP(Xn−1 = sn−1) > 0.
Definition: Time-homogeneousA discrete-time Markov process Xn | n ∈ Nis time-homogeneous if
P(Xn+1 = s ′ | Xn = s) = P(X1 = s ′ | X0 = s)
for all n > 1 and s, s ′ ∈ S with P(X0 = s) > 0.
23 / 84
![Page 31: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/31.jpg)
Discrete-time Stochastic Processes - Construction (6)
Weather Example: Option 3 - one past dayI a distribution ps′ over sun, rain for
each s ′ ∈ S ,I for each i ≥ 1 and s0 · · · si ∈ S i+1
P(C (s0 . . . si ) | C (s0 . . . si−1)) = psi−1 (si )
I a distribution π over sun, rain suchthat P(C (s0)) = π(s0).
Overly restrictive, isn’t it?
Not really – one only needs to extend the state spaceI S = 1, . . . , 365 × sun, rain × sun, rain,I now each state encodes current day of the year, current
weather, and weather yesterday,I we can define over S a time-homogeneous Markov process
based on both Options 1 & 2 given earlier.
24 / 84
![Page 32: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/32.jpg)
Discrete-time Stochastic Processes - Construction (6)
Weather Example: Option 3 - one past dayI a distribution ps′ over sun, rain for
each s ′ ∈ S ,I for each i ≥ 1 and s0 · · · si ∈ S i+1
P(C (s0 . . . si ) | C (s0 . . . si−1)) = psi−1 (si )
I a distribution π over sun, rain suchthat P(C (s0)) = π(s0).
Overly restrictive, isn’t it?
Not really – one only needs to extend the state spaceI S = 1, . . . , 365 × sun, rain × sun, rain,I now each state encodes current day of the year, current
weather, and weather yesterday,I we can define over S a time-homogeneous Markov process
based on both Options 1 & 2 given earlier.
24 / 84
![Page 33: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/33.jpg)
Discrete-time Stochastic Processes - Construction (6)
Weather Example: Option 3 - one past dayI a distribution ps′ over sun, rain for
each s ′ ∈ S ,I for each i ≥ 1 and s0 · · · si ∈ S i+1
P(C (s0 . . . si ) | C (s0 . . . si−1)) = psi−1 (si )
I a distribution π over sun, rain suchthat P(C (s0)) = π(s0).
Overly restrictive, isn’t it?
Not really – one only needs to extend the state spaceI S = 1, . . . , 365 × sun, rain × sun, rain,I now each state encodes current day of the year, current
weather, and weather yesterday,I we can define over S a time-homogeneous Markov process
based on both Options 1 & 2 given earlier. 24 / 84
![Page 34: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/34.jpg)
Discrete-time Markov ChainsDTMC
25 / 84
![Page 35: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/35.jpg)
DTMC - Relation of Definitions
Stochastic process → Graph basedGiven a discrete-time homogeneous Markov process X (n) | n ∈ N
I with state space S ,I defined on a probability space (Ω,F ,P)
we take over the state space S and defineI P(s, s ′) = P(Xn = s ′ | Xn−1 = s) for an arbitrary n ∈ N andI π0(s) = P(X0 = s).
Graph based → stochastic processGiven a DTMC (S ,P, π0), we set Ω to S∞, F to the smallestσ-Algebra containing all cylinder sets and
P(C (s0 . . . sn)) = π0(s0) ·∏
1≤i≤n
P(si−1, si )
which uniquely defines the probability function P on F .
26 / 84
![Page 36: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/36.jpg)
DTMC - Conditional Probability and Expectation
Let (S ,P, π0) be a DTMC. We denote byI Ps the probability function of DTMC (S ,P, δs) where
δs(s ′) =
1 if s ′ = s
0 otherwise
I Es the expectation with respect to Ps
27 / 84
![Page 37: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/37.jpg)
Analysis questions
I Transient analysisI Steady-state analysisI RewardsI ReachabilityI Probabilistic logics
28 / 84
![Page 38: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/38.jpg)
DTMC - Transient Analysis
29 / 84
![Page 39: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/39.jpg)
DTMC - Transient Analysis - Example (1)
Example: Gambling with a Limit
0 10 20 30 40
1
1/2
1/2
1/2
1/2
1/2
1/2
1
What is the probability of being in state 0 after 3 steps?
30 / 84
![Page 40: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/40.jpg)
DTMC - Transient Analysis - n-step Probabilities
Definition:Given a DTMC (S ,P, π0), we assume w.l.o.g. S = 0, 1, . . . and writepij = P(i , j). Further, we have
I P(1) = P = (pij) is the 1-step transition matrixI P(n) = (p
(n)ij ) denotes the n-step transition matrix with
p(n)ij = P(Xn = j | X0 = i) (= P(Xk+n = j | Xk = i)).
How can we compute these probabilities?
31 / 84
![Page 41: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/41.jpg)
DTMC - Transient Analysis - Chapman-Kolmogorov
Definition: Chapman-Kolmogorov EquationApplication of the law of total probability to the n-step transitionprobabilities p
(n)ij results in the Chapman-Kolmogorov Equation
p(n)ij =
∑h∈S
p(m)ih p
(n−m)hj ∀0 < m < n.
Consequently, we have P(n) = PP(n−1) = · · · = Pn.
Definition: Transient Probability DistributionThe transient probability distribution at time n > 0 is defined by
πn = π0Pn = πn−1P.
32 / 84
![Page 42: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/42.jpg)
DTMC - Transient Analysis - Chapman-Kolmogorov
Definition: Chapman-Kolmogorov EquationApplication of the law of total probability to the n-step transitionprobabilities p
(n)ij results in the Chapman-Kolmogorov Equation
p(n)ij =
∑h∈S
p(m)ih p
(n−m)hj ∀0 < m < n.
Consequently, we have P(n) = PP(n−1) = · · · = Pn.
Definition: Transient Probability DistributionThe transient probability distribution at time n > 0 is defined by
πn = π0Pn = πn−1P.
32 / 84
![Page 43: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/43.jpg)
DTMC - Transient Analysis - Example (2)
0 10 20 30 40
1
1/2
1/2
1/2
1/2
1/2
1/2
1
Example:
P =
1 0 0 0 0
0.5 0 0.5 0 00 0.5 0 0.5 00 0 0.5 0 0.50 0 0 0 1
P2 =
1 0 0 0 0
0.5 0.25 0 0.25 00.25 0 0.5 0 0.25
0 0.25 0 0.25 0.50 0 0 0 1
I For π0 =
[0 0 1 0 0
], π2 = π0P2 =
[0.25 0 0.5 0 0.25
].
I For, π0 =[0.4 0 0 0 0.6
], π2 = π0P2 =
[0.4 0 0 0 0.6
].
Actually, πn =[0.4 0 0 0 0.6
]for all n ∈ N!
Are there other “stable” distributions?
33 / 84
![Page 44: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/44.jpg)
DTMC - Transient Analysis - Example (2)
0 10 20 30 40
1
1/2
1/2
1/2
1/2
1/2
1/2
1
Example:
P =
1 0 0 0 0
0.5 0 0.5 0 00 0.5 0 0.5 00 0 0.5 0 0.50 0 0 0 1
P2 =
1 0 0 0 0
0.5 0.25 0 0.25 00.25 0 0.5 0 0.25
0 0.25 0 0.25 0.50 0 0 0 1
I For π0 =
[0 0 1 0 0
], π2 = π0P2 =
[0.25 0 0.5 0 0.25
].
I For, π0 =[0.4 0 0 0 0.6
], π2 = π0P2 =
[0.4 0 0 0 0.6
].
Actually, πn =[0.4 0 0 0 0.6
]for all n ∈ N!
Are there other “stable” distributions?33 / 84
![Page 45: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/45.jpg)
DTMC - Steady State Analysis
34 / 84
![Page 46: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/46.jpg)
DTMC - Steady State Analysis - Definitions
Definition: Stationary DistributionA distribution π is stationary if
π = πP.
Stationary distribution is generally not unique.
Definition: Limiting Distribution
π∗ := limn→∞
πn = limn→∞
π0Pn = π0 limn→∞
Pn = π0P∗.
The limit can depend on π0 and does not need to exist.
Connection between stationary and limiting?
35 / 84
![Page 47: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/47.jpg)
DTMC - Steady State Analysis - Definitions
Definition: Stationary DistributionA distribution π is stationary if
π = πP.
Stationary distribution is generally not unique.
Definition: Limiting Distribution
π∗ := limn→∞
πn = limn→∞
π0Pn = π0 limn→∞
Pn = π0P∗.
The limit can depend on π0 and does not need to exist.
Connection between stationary and limiting?
35 / 84
![Page 48: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/48.jpg)
DTMC - Steady State Analysis - Definitions
Definition: Stationary DistributionA distribution π is stationary if
π = πP.
Stationary distribution is generally not unique.
Definition: Limiting Distribution
π∗ := limn→∞
πn = limn→∞
π0Pn = π0 limn→∞
Pn = π0P∗.
The limit can depend on π0 and does not need to exist.
Connection between stationary and limiting?
35 / 84
![Page 49: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/49.jpg)
DTMC - Steady-State Analysis - Periodicity
Example: Gambling with Social Guarantees
0 10 20 30 401
1/2
1/2
1/2
1/2
1/2
1/2
1
What are the stationary and limiting distributions?
Definition: PeriodicityThe period of a state i is defined as
di = gcdn | pnii > 0.
A state i is called aperiodic if di = 1 and periodic with period diotherwise. A Markov chain is aperiodic if all states are aperiodic.
LemmaIn a finite aperiodic Markov chain, the limiting distribution exists.
36 / 84
![Page 50: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/50.jpg)
DTMC - Steady-State Analysis - Periodicity
Example: Gambling with Social Guarantees
0 10 20 30 401
1/2
1/2
1/2
1/2
1/2
1/2
1
What are the stationary and limiting distributions?
Definition: PeriodicityThe period of a state i is defined as
di = gcdn | pnii > 0.
A state i is called aperiodic if di = 1 and periodic with period diotherwise. A Markov chain is aperiodic if all states are aperiodic.
LemmaIn a finite aperiodic Markov chain, the limiting distribution exists.
36 / 84
![Page 51: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/51.jpg)
DTMC - Steady-State Analysis - Irreducibility (1)
Example
0 10 20 30 40
1
1/2
1/2
1/2
1/2
1/2
1/2
1
37 / 84
![Page 52: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/52.jpg)
DTMC - Steady-State Analysis - Irreducibility (2)
Definition:A DTMC is called irreducible if for all states i , j ∈ S we have pnij > 0for some n ≥ 1.
LemmaIn an aperiodic and irreducible Markov chain, the limitingdistribution exists and does not depend on π0.
Examples
0 1
1 1
0 1
1
1
Zhang (Saarland University, Germany) Quantitative Model Checking August 26th , 2009 24 / 1
Examples
0 1
1 1
0 1
1
1
Zhang (Saarland University, Germany) Quantitative Model Checking August 26th , 2009 24 / 1
Examples
0 1
11
0 1
34
121
4
12
Zhang (Saarland University, Germany) Quantitative Model Checking August 26th , 2009 25 / 1
Examples
0 1
11
0 1
34
121
4
12
Zhang (Saarland University, Germany) Quantitative Model Checking August 26th , 2009 25 / 1
!
!
!
38 / 84
![Page 53: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/53.jpg)
DTMC - Steady-State Analysis - Irreducibility (3)
0 10 20 30 40
1/2
1/2
1/2
1/2
1/2
1/2
1/2
1/2
1/2
1/2
What is the stationary / limiting distribution?
0 10 20 30 . . .
1/2
1/2
1/2
1/2
1/2
1/2
1/2
1/2
1/2
LemmaIn a finite aperiodic and irreducible Markov chain, the limitingdistribution exists, does not depend on π0, and equals the uniquestationary distribution.
39 / 84
![Page 54: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/54.jpg)
DTMC - Steady-State Analysis - Irreducibility (3)
0 10 20 30 40
1/2
1/2
1/2
1/2
1/2
1/2
1/2
1/2
1/2
1/2
What is the stationary / limiting distribution?
0 10 20 30 . . .
1/2
1/2
1/2
1/2
1/2
1/2
1/2
1/2
1/2
LemmaIn a finite aperiodic and irreducible Markov chain, the limitingdistribution exists, does not depend on π0, and equals the uniquestationary distribution.
39 / 84
![Page 55: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/55.jpg)
DTMC - Steady-State Analysis - Irreducibility (3)
0 10 20 30 40
1/2
1/2
1/2
1/2
1/2
1/2
1/2
1/2
1/2
1/2
What is the stationary / limiting distribution?
0 10 20 30 . . .
1/2
1/2
1/2
1/2
1/2
1/2
1/2
1/2
1/2
LemmaIn a finite aperiodic and irreducible Markov chain, the limitingdistribution exists, does not depend on π0, and equals the uniquestationary distribution.
39 / 84
![Page 56: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/56.jpg)
DTMC - Steady-State Analysis - Recurrence (1)
Definition:Let f (n)
ij = P(Xn = j ∧ ∀1 ≤ k < n : Xk 6= j | X0 = i) for n ≥ 1 be then-step hitting probability. The hitting probability is defined as
fij =∞∑n=1
f(n)ij
and a state i is calledI transient if fii < 1 andI recurrent if fii = 1.
40 / 84
![Page 57: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/57.jpg)
DTMC - Steady-State Analysis - Recurrence (2)
Definition:Denoting expectation mij =
∑∞n=1 n · f
(n)ij , a recurrent state i is called
I positive recurrent or recurrent non-null if mii <∞ andI recurrent null if mii =∞.
LemmaThe states of an irreducible DTMC are all of the same type, i.e.,
I all periodic orI all aperiodic and transient orI all aperiodic and recurrent null orI all aperiodic and recurrent non-null.
41 / 84
![Page 58: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/58.jpg)
DTMC - Steady-State Analysis - Ergodicity
Definition: ErgodicityA DTMC is ergodic if all its states are irreducible, aperiodic andrecurrent non-null.
TheoremIn an ergodic Markov chain, the limiting distribution exists, does notdepend on π0, and equals the unique stationary distribution.
As a consequence, the steady-state distribution can be computed bysolving the equation system
π = πP,∑x∈S
πs = 1.
Note: The Lemma for finite DTMC follows from the theorem as everyirreducible finite DTMC is positive recurrent.
42 / 84
![Page 59: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/59.jpg)
DTMC - Steady-State Analysis - Ergodicity
Example: Unbounded Gambling with House Edge
0 10 20 30 . . .
1 − p
p
1 − p
p
1 − p
p
1 − p
1
1 − p
The DTMC is only ergodic for p ∈ [0, 0.5).
43 / 84
![Page 60: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/60.jpg)
DTMC - Rewards
44 / 84
![Page 61: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/61.jpg)
DTMC - Rewards - Definitions
DefinitionA reward Markov chain is a tuple (S ,P, π0, r) where (S ,P, π0) is aMarkov chain and r : S → Z is a reward function.
Every run ρ = s0, s1, . . . induces a sequence of values r(s0), r(s1), . . .
Value of the whole run can be defined astotal reward ∑T
i=1 r(si ) But what if T =∞?discounted reward∑∞
i=1 λi r(si ) for some 0 < λ < 1
average rewardlimn→∞
1n
∑ni=0 r(si )
also called long-run average or mean payoff
DefinitionThe expected average reward is
EAR := limn→∞
1
n
n∑i=0
E[r(Xi )]
45 / 84
![Page 62: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/62.jpg)
DTMC - Rewards - Definitions
DefinitionA reward Markov chain is a tuple (S ,P, π0, r) where (S ,P, π0) is aMarkov chain and r : S → Z is a reward function.
Every run ρ = s0, s1, . . . induces a sequence of values r(s0), r(s1), . . .
Value of the whole run can be defined as
total reward ∑Ti=1 r(si ) But what if T =∞?
discounted reward∑∞i=1 λ
i r(si ) for some 0 < λ < 1average reward
limn→∞1n
∑ni=0 r(si )
also called long-run average or mean payoff
DefinitionThe expected average reward is
EAR := limn→∞
1
n
n∑i=0
E[r(Xi )]
45 / 84
![Page 63: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/63.jpg)
DTMC - Rewards - Definitions
DefinitionA reward Markov chain is a tuple (S ,P, π0, r) where (S ,P, π0) is aMarkov chain and r : S → Z is a reward function.
Every run ρ = s0, s1, . . . induces a sequence of values r(s0), r(s1), . . .
Value of the whole run can be defined astotal reward ∑T
i=1 r(si )
But what if T =∞?discounted reward∑∞
i=1 λi r(si ) for some 0 < λ < 1
average rewardlimn→∞
1n
∑ni=0 r(si )
also called long-run average or mean payoff
DefinitionThe expected average reward is
EAR := limn→∞
1
n
n∑i=0
E[r(Xi )]
45 / 84
![Page 64: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/64.jpg)
DTMC - Rewards - Definitions
DefinitionA reward Markov chain is a tuple (S ,P, π0, r) where (S ,P, π0) is aMarkov chain and r : S → Z is a reward function.
Every run ρ = s0, s1, . . . induces a sequence of values r(s0), r(s1), . . .
Value of the whole run can be defined astotal reward ∑T
i=1 r(si ) But what if T =∞?
discounted reward∑∞i=1 λ
i r(si ) for some 0 < λ < 1average reward
limn→∞1n
∑ni=0 r(si )
also called long-run average or mean payoff
DefinitionThe expected average reward is
EAR := limn→∞
1
n
n∑i=0
E[r(Xi )]
45 / 84
![Page 65: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/65.jpg)
DTMC - Rewards - Definitions
DefinitionA reward Markov chain is a tuple (S ,P, π0, r) where (S ,P, π0) is aMarkov chain and r : S → Z is a reward function.
Every run ρ = s0, s1, . . . induces a sequence of values r(s0), r(s1), . . .
Value of the whole run can be defined astotal reward ∑T
i=1 r(si ) But what if T =∞?discounted reward∑∞
i=1 λi r(si ) for some 0 < λ < 1
average rewardlimn→∞
1n
∑ni=0 r(si )
also called long-run average or mean payoff
DefinitionThe expected average reward is
EAR := limn→∞
1
n
n∑i=0
E[r(Xi )]
45 / 84
![Page 66: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/66.jpg)
DTMC - Rewards - Definitions
DefinitionA reward Markov chain is a tuple (S ,P, π0, r) where (S ,P, π0) is aMarkov chain and r : S → Z is a reward function.
Every run ρ = s0, s1, . . . induces a sequence of values r(s0), r(s1), . . .
Value of the whole run can be defined astotal reward ∑T
i=1 r(si ) But what if T =∞?discounted reward∑∞
i=1 λi r(si ) for some 0 < λ < 1
average rewardlimn→∞
1n
∑ni=0 r(si )
also called long-run average or mean payoff
DefinitionThe expected average reward is
EAR := limn→∞
1
n
n∑i=0
E[r(Xi )]
45 / 84
![Page 67: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/67.jpg)
DTMC - Rewards - Definitions
DefinitionA reward Markov chain is a tuple (S ,P, π0, r) where (S ,P, π0) is aMarkov chain and r : S → Z is a reward function.
Every run ρ = s0, s1, . . . induces a sequence of values r(s0), r(s1), . . .
Value of the whole run can be defined astotal reward ∑T
i=1 r(si ) But what if T =∞?discounted reward∑∞
i=1 λi r(si ) for some 0 < λ < 1
average rewardlimn→∞
1n
∑ni=0 r(si )
also called long-run average or mean payoff
DefinitionThe expected average reward is
EAR := limn→∞
1
n
n∑i=0
E[r(Xi )]
45 / 84
![Page 68: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/68.jpg)
DTMC - Rewards - Solution Sketch
Definition: Time-average Distribution
π = limn→∞
1
n
n∑i=0
πi .
π(s) expresses the ratio of time spent in s on the long run.
Lemma1. E[r(Xi )] =
∑s∈S πi (s) · r(s).
2. If π exists then EAR =∑
s∈S π(s) · r(s).3. If limiting distribution exists, it coincides with π.
Algortithm1. Compute π (or limiting distribution if possible).1
2. Return∑
s∈S π(s) · r(s).
1More details later for Markov decision processes.46 / 84
![Page 69: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/69.jpg)
DTMC - Rewards - Solution Sketch
Definition: Time-average Distribution
π = limn→∞
1
n
n∑i=0
πi .
π(s) expresses the ratio of time spent in s on the long run.
Lemma1. E[r(Xi )] =
∑s∈S πi (s) · r(s).
2. If π exists then EAR =∑
s∈S π(s) · r(s).3. If limiting distribution exists, it coincides with π.
Algortithm1. Compute π (or limiting distribution if possible).1
2. Return∑
s∈S π(s) · r(s).
1More details later for Markov decision processes.46 / 84
![Page 70: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/70.jpg)
DTMC - Rewards - Solution Sketch
Definition: Time-average Distribution
π = limn→∞
1
n
n∑i=0
πi .
π(s) expresses the ratio of time spent in s on the long run.
Lemma1. E[r(Xi )] =
∑s∈S πi (s) · r(s).
2. If π exists then EAR =∑
s∈S π(s) · r(s).3. If limiting distribution exists, it coincides with π.
Algortithm1. Compute π (or limiting distribution if possible).1
2. Return∑
s∈S π(s) · r(s).
1More details later for Markov decision processes.46 / 84
![Page 71: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/71.jpg)
DTMC - Reachability
47 / 84
![Page 72: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/72.jpg)
DTMC - Reachability
Definition: ReachabilityGiven a DTMC (S ,P, π0), what is the probability of eventuallyreaching a set of goal states B ⊆ S?
S
Bsxs
Let x(s) denote Ps(♦B) where ♦B = s0s1 · · · | ∃i : si ∈ B. ThenI s ∈ B : x(s) =
1
I s ∈ S \ B : x(s) =
∑t∈S\B P(s, t)x(t) +
∑u∈B P(s, u).
48 / 84
![Page 73: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/73.jpg)
DTMC - Reachability
Definition: ReachabilityGiven a DTMC (S ,P, π0), what is the probability of eventuallyreaching a set of goal states B ⊆ S?
S
Bsxs
Let x(s) denote Ps(♦B) where ♦B = s0s1 · · · | ∃i : si ∈ B. ThenI s ∈ B : x(s) = 1
I s ∈ S \ B : x(s) =∑
t∈S\B P(s, t)x(t) +∑
u∈B P(s, u).
48 / 84
![Page 74: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/74.jpg)
DTMC - Reachability
Lemma (Reachability Matrix Form)Given a DTMC (S ,P, π0), the column vector x = (x(s))s∈S\B ofprobabilities x(s) = Ps(♦B) satisfies the constraint
x = Ax + b,
where matrix A is the submatrix of P for states S \ B andb = (b(s))s∈S\B is the column vector with b(s) =
∑u∈B P(s, u).
49 / 84
![Page 75: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/75.jpg)
DTMC - Reachability
Example:
s0
s1s2 s3
0.5
0.5
0.5
0.250.25
1 1
B = s3
0000
0.5 0.5 00.5 0.25 0.250 1 00 0 1
P =
A b
The vector x =[x0 x1 x2
]T=[0.25 0.5 0
]T satisfies theequation system x = Ax + b.
Is it the only solution?I No! Consider, e.g.,
[0.55 0.7 0.4
]or[1 1 1
]T .I While reaching the goal, such bad states need to be avoided.I We first generalise such “avoiding”.
50 / 84
![Page 76: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/76.jpg)
DTMC - Reachability
Example:
s0
s1s2 s3
0.5
0.5
0.5
0.250.25
1 1
B = s3
0000
0.5 0.5 00.5 0.25 0.250 1 00 0 1
P =
A b
The vector x =[x0 x1 x2
]T=[0.25 0.5 0
]T satisfies theequation system x = Ax + b.
Is it the only solution?
I No! Consider, e.g.,[0.55 0.7 0.4
]or[1 1 1
]T .I While reaching the goal, such bad states need to be avoided.I We first generalise such “avoiding”.
50 / 84
![Page 77: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/77.jpg)
DTMC - Reachability
Example:
s0
s1s2 s3
0.5
0.5
0.5
0.250.25
1 1
B = s3
0000
0.5 0.5 00.5 0.25 0.250 1 00 0 1
P =
A b
The vector x =[x0 x1 x2
]T=[0.25 0.5 0
]T satisfies theequation system x = Ax + b.
Is it the only solution?I No! Consider, e.g.,
[0.55 0.7 0.4
]or[1 1 1
]T .I While reaching the goal, such bad states need to be avoided.I We first generalise such “avoiding”.
50 / 84
![Page 78: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/78.jpg)
DTMC - Conditional Reachability
Definition:Let B,C ⊆ S . The (unbounded) probability of reaching B from state sunder the condition that C is not left before is defined asPs(C U B) where
C U B = s0s1 · · · | ∃i : si ∈ B ∧ ∀j < i : sj ∈ C.
The probability of reaching B from state s within n steps under thecondition that C is not left before is defined as Ps(C U ≤n B) where
C U ≤n B = s0s1 · · · | ∃i ≤ n : si ∈ B ∧ ∀j < i : sj ∈ C.
What is the equation system for these probabilities?
51 / 84
![Page 79: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/79.jpg)
DTMC - Conditional Reachability
Definition:Let B,C ⊆ S . The (unbounded) probability of reaching B from state sunder the condition that C is not left before is defined asPs(C U B) where
C U B = s0s1 · · · | ∃i : si ∈ B ∧ ∀j < i : sj ∈ C.
The probability of reaching B from state s within n steps under thecondition that C is not left before is defined as Ps(C U ≤n B) where
C U ≤n B = s0s1 · · · | ∃i ≤ n : si ∈ B ∧ ∀j < i : sj ∈ C.
What is the equation system for these probabilities?
51 / 84
![Page 80: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/80.jpg)
DTMC - Conditional Reachability - Solution
Let S=0 = s | Ps(C U B) = 0 and S? = S \ (S=0 ∪ B).
Theorem:The column vector x = (x(s))s∈S?
of probabilities x(s) = Ps(C U B)is the unique solution of the equation system
x = Ax + b,
where A = (P(s, t))s,t∈S?, b = (b(s))s∈S?
with b(s) =∑
u∈B P(s, u).
Furthermore, for x0 = (0)s∈S?and xi = Axi−1 + b for any i ≥ 1,
1. xn(s) = Ps(C U ≤n B) for s ∈ S?,2. xi is increasing, and3. x = limn→∞ xn.
52 / 84
![Page 81: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/81.jpg)
DTMC - Conditional Reachability - Solution
Let S=0 = s | Ps(C U B) = 0 and S? = S \ (S=0 ∪ B).
Theorem:The column vector x = (x(s))s∈S?
of probabilities x(s) = Ps(C U B)is the unique solution of the equation system
x = Ax + b,
where A = (P(s, t))s,t∈S?, b = (b(s))s∈S?
with b(s) =∑
u∈B P(s, u).
Furthermore, for x0 = (0)s∈S?and xi = Axi−1 + b for any i ≥ 1,
1. xn(s) = Ps(C U ≤n B) for s ∈ S?,2. xi is increasing, and3. x = limn→∞ xn.
52 / 84
![Page 82: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/82.jpg)
DTMC - Conditional Reachability - Proof
Proof Sketch:I (xs)x∈S?
is a solution: by inserting into definition.I Unique solution: By contradiction. Assume y is another solution,
then x− y = A(x− y). One can show that A− I is invertible, thus(A− I)(x− y) = 0 yields x− y = (A− I)−10 = 0 and finally x = y 2.
Furthermore,1. From the definitions, by straightforward induction.2. From 1. since C U ≤n B ⊆ C U ≤n+1 B .3. Since C U B =
⋃n∈N C U ≤n B .
2cf. page 766 of Principles of Model Checking53 / 84
![Page 83: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/83.jpg)
Algorithmic aspects
54 / 84
![Page 84: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/84.jpg)
Algorithmic Aspects - Summary of Equation Systems
Equation SystemsI Transient analysis: πn = π0Pn = πn−1P
I Steady-state analysis: πP = π, π · 1 =∑
s∈S π(s) = 1 (ergodic)I Reachability: x = Ax + b (with (x(s))s∈S?
)
Solution Techniques1. Analytic solution, e.g. by Gaussian elimination2. Iterative power method (πn → π and xn → x for n→∞)3. Iterative methods for solving large systems of linear equations,
e.g. Jacobi, Gauss-Seidel
Missing piecesa. finding out whether a DTMC is ergodic,b. computing S? = S \ s | Ps(♦ B) = 0,c. efficient representation of P.
55 / 84
![Page 85: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/85.jpg)
Algorithmic Aspects: a. Ergodicity of finite DTMC (1)
Ergodicity = Irreducibility + Aperidocity + P. RecurrenceI A DTMC is called irreducible if for all states i , j ∈ S we have
pnij > 0 for some n ≥ 1.I A state i is called aperiodic if gcdn | pnii > 0 = 1.I A state i is called positive recurrent if fii = 1 and mii <∞.
How do we tell that a finite DTMC is ergodic?
By analysis of the induced graph!For a DTMC (S ,P, π(0)) we define the induced directed graph (S ,E )with E = (s, s ′) | P(s, s ′) > 0.Recall:
I A directed graph is called strongly connected if there is a pathfrom each vertex to every other vertex.
I Strongly connected components (SCC) are its maximal stronglyconnected subgraphs.
I A SCC T is bottom (BSCC) if no s 6∈ T is reachable from T .
56 / 84
![Page 86: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/86.jpg)
Algorithmic Aspects: a. Ergodicity of finite DTMC (1)
Ergodicity = Irreducibility + Aperidocity + P. RecurrenceI A DTMC is called irreducible if for all states i , j ∈ S we have
pnij > 0 for some n ≥ 1.I A state i is called aperiodic if gcdn | pnii > 0 = 1.I A state i is called positive recurrent if fii = 1 and mii <∞.
How do we tell that a finite DTMC is ergodic?By analysis of the induced graph!For a DTMC (S ,P, π(0)) we define the induced directed graph (S ,E )with E = (s, s ′) | P(s, s ′) > 0.Recall:
I A directed graph is called strongly connected if there is a pathfrom each vertex to every other vertex.
I Strongly connected components (SCC) are its maximal stronglyconnected subgraphs.
I A SCC T is bottom (BSCC) if no s 6∈ T is reachable from T .56 / 84
![Page 87: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/87.jpg)
Algorithmic Aspects: a. Ergodicity of finite DTMC (2)
Ergodicity = Irreducibility + Aperidocity + P. RecurrenceI A DTMC is called irreducible if for all states i , j ∈ S we have
pnij > 0 for some n ≥ 1.I A state i is called aperiodic if gcdn | pnii > 0 = 1.I A state i is called positive recurrent if fii = 1 and mii <∞.
Theorem:For finite DTMCs, it holds that:
I The DTMC is irreducible iff the induced graph is stronglyconnected.
I A state in a BSCC is aperiodic iff the BSCC is aperiodic, i.e. thegreatest common divisor of the lengths of all its cycles is 1.
I A state is positive recurrent iff it belongs to a BSCC otherwise itis transient.
57 / 84
![Page 88: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/88.jpg)
Algorithmic Aspects: a. Ergodicity of finite DTMC (2)
Ergodicity = Irreducibility + Aperidocity + P. RecurrenceI A DTMC is called irreducible if for all states i , j ∈ S we have
pnij > 0 for some n ≥ 1.I A state i is called aperiodic if gcdn | pnii > 0 = 1.I A state i is called positive recurrent if fii = 1 and mii <∞.
Theorem:For finite DTMCs, it holds that:
I The DTMC is irreducible iff the induced graph is stronglyconnected.
I A state in a BSCC is aperiodic iff the BSCC is aperiodic, i.e. thegreatest common divisor of the lengths of all its cycles is 1.
I A state is positive recurrent iff it belongs to a BSCC otherwise itis transient.
57 / 84
![Page 89: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/89.jpg)
Algorithmic Aspects: a. Ergodicity of finite DTMC (2)
Ergodicity = Irreducibility + Aperidocity + P. RecurrenceI A DTMC is called irreducible if for all states i , j ∈ S we have
pnij > 0 for some n ≥ 1.I A state i is called aperiodic if gcdn | pnii > 0 = 1.I A state i is called positive recurrent if fii = 1 and mii <∞.
Theorem:For finite DTMCs, it holds that:
I The DTMC is irreducible iff the induced graph is stronglyconnected.
I A state in a BSCC is aperiodic iff the BSCC is aperiodic, i.e. thegreatest common divisor of the lengths of all its cycles is 1.
I A state is positive recurrent iff it belongs to a BSCC otherwise itis transient.
57 / 84
![Page 90: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/90.jpg)
Algorithmic Aspects: a. Ergodicity of finite DTMC (2)
Ergodicity = Irreducibility + Aperidocity + P. RecurrenceI A DTMC is called irreducible if for all states i , j ∈ S we have
pnij > 0 for some n ≥ 1.I A state i is called aperiodic if gcdn | pnii > 0 = 1.I A state i is called positive recurrent if fii = 1 and mii <∞.
Theorem:For finite DTMCs, it holds that:
I The DTMC is irreducible iff the induced graph is stronglyconnected.
I A state in a BSCC is aperiodic iff the BSCC is aperiodic, i.e. thegreatest common divisor of the lengths of all its cycles is 1.
I A state is positive recurrent iff it belongs to a BSCC otherwise itis transient.
57 / 84
![Page 91: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/91.jpg)
Algorithmic Aspects: a. Ergodicity of finite DTMC (3)How to check: is gcd of the lengths of all cycles of a stronglyconnected graph 1?
I gcdn ≥ 1 | ∃s : Pn(s, s) > 0 = 1I in time O(n + m)? By the following DFS-based procedure:
Algorithm: PERIOD(vertex v , unsigned level : init 0)1 global period : init 0;2 if period = 1 then3 return4 end5 if v is unmarked then6 mark v ;7 vlevel = level ;8 for v ′ ∈ out(v) do9 PERIOD(v ′,level + 1)
10 end11 else12 period = gcd(period , level − vlevel);13 end
58 / 84
![Page 92: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/92.jpg)
Algorithmic Aspects: a. Ergodicity of finite DTMC (3)How to check: is gcd of the lengths of all cycles of a stronglyconnected graph 1?
I gcdn ≥ 1 | ∃s : Pn(s, s) > 0 = 1
I in time O(n + m)? By the following DFS-based procedure:Algorithm: PERIOD(vertex v , unsigned level : init 0)
1 global period : init 0;2 if period = 1 then3 return4 end5 if v is unmarked then6 mark v ;7 vlevel = level ;8 for v ′ ∈ out(v) do9 PERIOD(v ′,level + 1)
10 end11 else12 period = gcd(period , level − vlevel);13 end
58 / 84
![Page 93: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/93.jpg)
Algorithmic Aspects: a. Ergodicity of finite DTMC (3)How to check: is gcd of the lengths of all cycles of a stronglyconnected graph 1?
I gcdn ≥ 1 | ∃s : Pn(s, s) > 0 = 1I in time O(n + m)?
By the following DFS-based procedure:Algorithm: PERIOD(vertex v , unsigned level : init 0)
1 global period : init 0;2 if period = 1 then3 return4 end5 if v is unmarked then6 mark v ;7 vlevel = level ;8 for v ′ ∈ out(v) do9 PERIOD(v ′,level + 1)
10 end11 else12 period = gcd(period , level − vlevel);13 end
58 / 84
![Page 94: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/94.jpg)
Algorithmic Aspects: a. Ergodicity of finite DTMC (3)How to check: is gcd of the lengths of all cycles of a stronglyconnected graph 1?
I gcdn ≥ 1 | ∃s : Pn(s, s) > 0 = 1I in time O(n + m)? By the following DFS-based procedure:
Algorithm: PERIOD(vertex v , unsigned level : init 0)1 global period : init 0;2 if period = 1 then3 return4 end5 if v is unmarked then6 mark v ;7 vlevel = level ;8 for v ′ ∈ out(v) do9 PERIOD(v ′,level + 1)
10 end11 else12 period = gcd(period , level − vlevel);13 end
58 / 84
![Page 95: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/95.jpg)
Algorithmic Aspects: b. Computing the set S?
We have S? = S \ (B ∪ S=0) where S=0 = s | Ps(♦ B) = 0.Hence,
s ∈ S=0 iff pnss′ = 0 for all n ≥ 1 and s ′ ∈ B.
This can be again easily checked from the induced graph:
LemmaWe have s ∈ S=0 iff there is no path from s to any state from B .
Proof.Easy from the fact that pnss′ > 0 iff there is a path of length n to s ′.
59 / 84
![Page 96: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/96.jpg)
Algorithmic Aspects: b. Computing the set S?
We have S? = S \ (B ∪ S=0) where S=0 = s | Ps(♦ B) = 0.Hence,
s ∈ S=0 iff pnss′ = 0 for all n ≥ 1 and s ′ ∈ B.
This can be again easily checked from the induced graph:
LemmaWe have s ∈ S=0 iff there is no path from s to any state from B .
Proof.Easy from the fact that pnss′ > 0 iff there is a path of length n to s ′.
59 / 84
![Page 97: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/97.jpg)
Algorithmic Aspects: c. Efficient Representations
Zeroconf as a Markov chain
s0 s1 s2 s3 s4
s5
s6
s7
s8
q p p p
p
1
1 ! q
1
1 ! p1 ! p
1 ! p
1 ! p
start
error
Zhang (Saarland University, Germany) Quantitative Model Checking August 24th , 2009 19 / 35
1. There are many 0 entries in the transition matrix.Sparse matrices offer a more concise storage.
2. There are many similar entries in the transition matrix.Multi-terminal binary decision diagrams offer a more concisestorage, using automata theory.
60 / 84
![Page 98: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/98.jpg)
Algorithmic Aspects: c. Efficient Representations
Zeroconf as a Markov chain
s0 s1 s2 s3 s4
s5
s6
s7
s8
q p p p
p
1
1 ! q
1
1 ! p1 ! p
1 ! p
1 ! p
start
error
Zhang (Saarland University, Germany) Quantitative Model Checking August 24th , 2009 19 / 35
1. There are many 0 entries in the transition matrix.Sparse matrices offer a more concise storage.
2. There are many similar entries in the transition matrix.Multi-terminal binary decision diagrams offer a more concisestorage, using automata theory.
60 / 84
![Page 99: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/99.jpg)
Algorithmic Aspects: c. Efficient Representations
Zeroconf as a Markov chain
s0 s1 s2 s3 s4
s5
s6
s7
s8
q p p p
p
1
1 ! q
1
1 ! p1 ! p
1 ! p
1 ! p
start
error
Zhang (Saarland University, Germany) Quantitative Model Checking August 24th , 2009 19 / 35
1. There are many 0 entries in the transition matrix.Sparse matrices offer a more concise storage.
2. There are many similar entries in the transition matrix.Multi-terminal binary decision diagrams offer a more concisestorage, using automata theory.
60 / 84
![Page 100: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/100.jpg)
DTMC - Probabilistic TemporalLogics for Specifying Complex
Properties
61 / 84
![Page 101: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/101.jpg)
Logics - Adding Labels to DTMC
Definition:A labeled DTMC is a tuple D = (S ,P, π0, L) with L : S → 2AP , where
I AP is a set of atomic propositions andI L is a labeling function, where L(s) specifies which properties
hold in state s ∈ S .
62 / 84
![Page 102: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/102.jpg)
Logics - Examples of Properties
HHH
OO
O
OOOH
HH
↑
↓ States and transitionsstate = configuration of the game;transition = rolling the dice and acting(randomly) based on the result.
State labelsI init, rwin, bwin, rkicked, bkicked, . . .I r30, r21, . . . ,I b30, b21,. . . ,
Examples of PropertiesI the game cannot return back to startI at any time, the game eventually ends with prob. 1
I at any time, the game ends within 100 dice rolls with prob. ≥ 0.5
I the probability of winning without ever being kicked out is ≤ 0.3
How to specify them formally? 63 / 84
![Page 103: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/103.jpg)
Logics - Temporal Logics - non-probabilistic (1)
Linear-time viewI corresponds to our (human) perception of timeI can specify properties of one concrete linear execution of the
systemExample: eventually red player is kicked out followed immediatelyby blue player being kicked out.
Branching-time viewI views future as a set of all possibilitiesI can specify properties of all executions from a given state –
specifies execution treesExample: in every computation it is always possible to return to theinitial state.
64 / 84
![Page 104: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/104.jpg)
Logics - Temporal Logics - non-probabilistic (2)
Linear Temporal Logic (LTL)Syntax for formulae specifying executions:
ψ = true | a | ψ ∧ ψ | ¬ψ | X ψ | ψ U ψ | F ψ | G ψExample: eventually red player is kicked out followed immediatelyby blue player being kicked out: F (rkicked ∧ X bkicked)Question: do all executions satisfy the given LTL formula?
Computation Tree Logic (CTL)Syntax for specifying states:
φ = true | a | φ ∧ φ | ¬φ | A ψ | E ψ
Syntax for specifying executions:
ψ = X φ | φ U φ | F φ | G φExample: in all computations it is always possible to return to initialstate: A G E F initQuestion: does the given state satisfy the given CTL state formula?
65 / 84
![Page 105: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/105.jpg)
Logics - LTL
Syntax ψ = true | a | ψ ∧ ψ | ¬ψ | X ψ | ψ U ψ.
Semantics (for a path ω = s0s1 · · · )I ω |= true (always),I ω |= a iff a ∈ L(s0),I ω |= ψ1 ∧ ψ2 iff ω |= ψ1 and ω |= ψ2,I ω |= ¬ψ iff ω 6|= ψ,I ω |= X ψ iff s1s2 · · · |= ψ,
Ψ
I ω |= ψ1 U ψ2 iff ∃i ≥ 0 : si si+1 · · · |= ψ2 and ∀j < i : sjsj+1 · · · |= ψ1.
Ψ1 · · · Ψ1 Ψ2
Syntactic sugarI F ψ ≡
true U ψ
I G ψ ≡
¬(true U ¬ψ) (≡ ¬F ¬ψ)
66 / 84
![Page 106: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/106.jpg)
Logics - LTL
Syntax ψ = true | a | ψ ∧ ψ | ¬ψ | X ψ | ψ U ψ.
Semantics (for a path ω = s0s1 · · · )I ω |= true (always),I ω |= a iff a ∈ L(s0),I ω |= ψ1 ∧ ψ2 iff ω |= ψ1 and ω |= ψ2,I ω |= ¬ψ iff ω 6|= ψ,I ω |= X ψ iff s1s2 · · · |= ψ,
Ψ
I ω |= ψ1 U ψ2 iff ∃i ≥ 0 : si si+1 · · · |= ψ2 and ∀j < i : sjsj+1 · · · |= ψ1.
Ψ1 · · · Ψ1 Ψ2
Syntactic sugarI F ψ ≡ true U ψ
I G ψ ≡ ¬(true U ¬ψ) (≡ ¬F ¬ψ)66 / 84
![Page 107: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/107.jpg)
Logics - CTL
SyntaxState formulae:
φ = true | a | φ ∧ φ | ¬φ | A ψ | E ψ
where ψ is a path formula.
Path formulae:
ψ = X φ | φ U φ
where φ is a state formula.
SemanticsFor a state s :
I s |= true (always),I s |= a iff a ∈ L(s),I s |= φ1 ∧ φ2 iff s |= φ1 and
s |= φ2,I s |= ¬φ iff s 6|= φ,I s |= Aψ iff ω |= ψ for all
paths ω = s0s1 · · · with s0 = s ,I s |= Eψ iff ω |= ψ for some
path ω = s0s1 · · · with s0 = s .
For a path ω = s0s1 · · · :I ω |= X φ iff
s1s2 · · · satisfies φ,
Φ
I ω |= φ1 U φ2 iff ∃i :si si+1 · · · |= φ2 and∀j < i : sjsj+1 · · · |= φ1.
Φ1 · · · Φ1 Φ2
67 / 84
![Page 108: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/108.jpg)
Logics - Temporal Logics - non-probabilistic (2)
Linear Temporal Logic (LTL)Syntax for formulae specifying executions:
ψ = true | a | ψ ∧ ψ | ¬ψ | X ψ | ψ U ψ | F ψ | G ψExample: eventually red player is kicked out followed immediatelyby blue player being kicked out: F (rkicked ∧ X bkicked)Question: do all executions satisfy the given LTL formula?
Computation Tree Logic (CTL)Syntax for specifying states:
φ = true | a | φ ∧ φ | ¬φ | A ψ | E ψ
Syntax for specifying executions:
ψ = X φ | φ U φ | F φ | G φExample: in all computations it is always possible to return to initialstate: A G E F initQuestion: does the given state satisfy the given CTL state formula?
68 / 84
![Page 109: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/109.jpg)
Logics - Temporal Logics - probabilistic
Linear Temporal Logic (LTL) + probabilitiesSyntax for formulae specifying executions:
ψ = true | a | ψ ∧ ψ | ¬ψ | X ψ | ψ U ψ | F ψ | G ψExample: with prob. ≥ 0.8, eventually red player is kicked outfollowed immediately by blue player being kicked out:
P(F (rkicked ∧ X bkicked)) ≥ 0.8
Question: is the formula satisfied by executions of given probability?
Probabilitic Computation Tree Logic (PCTL)Syntax for specifying states:
φ = true | a | φ ∧ φ | ¬φ | PJ ψ
Syntax for specifying executions:
ψ = X φ | φ U φ | φ U ≤kφ | F φ | G φExample: with prob. at least 0.5 the probability to return to initialstate is always at least 0.1: P≥0.5 G P≥0.1 F initQuestion: does the given state satisfy the given PCTL state formula?
69 / 84
![Page 110: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/110.jpg)
Logics - Temporal Logics - probabilistic
Linear Temporal Logic (LTL) + probabilitiesSyntax for formulae specifying executions:
ψ = true | a | ψ ∧ ψ | ¬ψ | X ψ | ψ U ψ | F ψ | G ψExample: with prob. ≥ 0.8, eventually red player is kicked outfollowed immediately by blue player being kicked out:
P(F (rkicked ∧ X bkicked)) ≥ 0.8
Question: is the formula satisfied by executions of given probability?
Probabilitic Computation Tree Logic (PCTL)Syntax for specifying states:
φ = true | a | φ ∧ φ | ¬φ | PJ ψ
Syntax for specifying executions:
ψ = X φ | φ U φ | φ U ≤kφ | F φ | G φExample: with prob. at least 0.5 the probability to return to initialstate is always at least 0.1: P≥0.5 G P≥0.1 F initQuestion: does the given state satisfy the given PCTL state formula?
69 / 84
![Page 111: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/111.jpg)
Logics - PCTL - Examples
Syntactic sugar:I φ1 ∨ φ2 ≡ ¬(¬φ1 ∧ ¬φ2), φ1 ⇒ φ2 ≡ ¬φ1 ∨ φ2, etc.I ≤ 0.5 denotes the interval [0, 0.5], = 1 denotes [1, 1], etc.
Examples:I A fair die: ∧
i∈1,...,6
P= 16(F i).
I The probability of winning ”Who wants to be a millionaire”without using any joker should be negligible:
P<1e−10(¬(J50% ∨ Jaudience ∨ Jtelephone) U win).
70 / 84
![Page 112: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/112.jpg)
Logics - PCTL - Semantics
SemanticsFor a state s :
I s |= true (always),I s |= a iff a ∈ L(s),I s |= φ1 ∧ φ2 iff s |= φ1 and
s |= φ2,I s |= ¬φ iff s 6|= φ,I
s |= PJ(ψ) iff Ps(Paths(ψ)) ∈ J
For a path ω = s0s1 · · · :I ω |= X φ iff s1s2 · · · satisfies φ,
!
!1 !1 !2. . . .
I ω |= φ1 U φ2 iff ∃i :si si+1 · · · |= φ2 and∀j < i : sjsj+1 · · · |= φ1.
!
!1 !1 !2. . . .
I ω |= φ1 U ≤nφ2 iff ∃i ≤ n :si si+1 · · · |= φ2 and∀j < i : sjsj+1 · · · |= φ1.
!
!1 !1 !2. . . .
71 / 84
![Page 113: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/113.jpg)
Logics - Examples of Properties
HHH
OO
O
OOOH
HH
↑
↓ Examples of Properties1. the game cannot return back to start2. at any time, the game eventually
ends with prob. 1
3. at any time, the game ends within100 dice rolls with prob. ≥ 0.5
4. the probability of winning withoutever being kicked out is ≤ 0.3
Formally
1. P(X G ¬init) = 1 (LTL + prob.)P=1(X P=0(G ¬init)) (PCTL)
2. P=1(G P=1(F (rwin ∨ bwin))) (PCTL)3. P=1(G P≥0.5(F ≤100(rwin ∨ bwin))) (PCTL)4. P((¬rkicked ∧ ¬bkicked) U (rwin ∨ bwin) ≤ 0.3 (LTL + prob.)
72 / 84
![Page 114: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/114.jpg)
Logics - Examples of Properties
HHH
OO
O
OOOH
HH
↑
↓ Examples of Properties1. the game cannot return back to start2. at any time, the game eventually
ends with prob. 1
3. at any time, the game ends within100 dice rolls with prob. ≥ 0.5
4. the probability of winning withoutever being kicked out is ≤ 0.3
Formally1. P(X G ¬init) = 1 (LTL + prob.)
P=1(X P=0(G ¬init)) (PCTL)2. P=1(G P=1(F (rwin ∨ bwin))) (PCTL)3. P=1(G P≥0.5(F ≤100(rwin ∨ bwin))) (PCTL)4. P((¬rkicked ∧ ¬bkicked) U (rwin ∨ bwin) ≤ 0.3 (LTL + prob.)
72 / 84
![Page 115: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/115.jpg)
PCTL Model Checking Algorithm
73 / 84
![Page 116: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/116.jpg)
PCTL Model Checking
Definition: PCTL Model CheckingLet D = (S ,P, π0, L) be a DTMC, Φ a PCTL state formula and s ∈ S .The model checking problem is to decide whether s |= Φ.
TheoremThe PCTL model checking problem can be decided in time polynomialin |D|, linear in |Φ|, and linear in the maximum step bound n.
74 / 84
![Page 117: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/117.jpg)
PCTL Model Checking - Algorithm - Outline (1)
Algorithm:Consider the bottom-up traversal of the parse tree of Φ:
I The leaves are a ∈ AP or true andI the inner nodes are:
I unary – labelled with the operator ¬ or PJ(X );I binary – labelled with an operator ∧, PJ( U ), or PJ( U ≤n).
Example: ¬a ∧ P≤0.2(¬b U P≥0.9(♦ c))
∧
aP!0.2( U )
¬
b
P"0.9( U )
true c
¬
Compute Sat(Ψ) = s ∈ S | s |= Ψ for each node Ψ of the tree in abottom-up fashion. Then s |= Φ iff s ∈ Sat(Φ).
75 / 84
![Page 118: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/118.jpg)
PCTL Model Checking - Algorithm - Outline (2)
“Base” of the algorithm:We need a procedure to compute Sat(Ψ) for Ψ of the form a or true:
LemmaI Sat(true) = S ,I Sat(a) = s | a ∈ L(s)
“Induction” step of the algorithm:We need a procedure to compute Sat(Ψ) for Ψ given the sets Sat(Ψ′)for all state sub-formulas Ψ′ of Ψ:
LemmaI Sat(Φ1 ∧ Φ2) =
I Sat(¬Φ) =
Sat(PJ(Φ)) = s | Ps(Paths(Φ)) ∈ J discussed on the next slide.
76 / 84
![Page 119: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/119.jpg)
PCTL Model Checking - Algorithm - Outline (2)
“Base” of the algorithm:We need a procedure to compute Sat(Ψ) for Ψ of the form a or true:
LemmaI Sat(true) = S ,I Sat(a) = s | a ∈ L(s)
“Induction” step of the algorithm:We need a procedure to compute Sat(Ψ) for Ψ given the sets Sat(Ψ′)for all state sub-formulas Ψ′ of Ψ:
LemmaI Sat(Φ1 ∧ Φ2) =
I Sat(¬Φ) =
Sat(PJ(Φ)) = s | Ps(Paths(Φ)) ∈ J discussed on the next slide.
76 / 84
![Page 120: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/120.jpg)
PCTL Model Checking - Algorithm - Outline (2)
“Base” of the algorithm:We need a procedure to compute Sat(Ψ) for Ψ of the form a or true:
LemmaI Sat(true) = S ,I Sat(a) = s | a ∈ L(s)
“Induction” step of the algorithm:We need a procedure to compute Sat(Ψ) for Ψ given the sets Sat(Ψ′)for all state sub-formulas Ψ′ of Ψ:
LemmaI Sat(Φ1 ∧ Φ2) =
I Sat(¬Φ) =
Sat(PJ(Φ)) = s | Ps(Paths(Φ)) ∈ J discussed on the next slide.
76 / 84
![Page 121: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/121.jpg)
PCTL Model Checking - Algorithm - Outline (2)
“Base” of the algorithm:We need a procedure to compute Sat(Ψ) for Ψ of the form a or true:
LemmaI Sat(true) = S ,I Sat(a) = s | a ∈ L(s)
“Induction” step of the algorithm:We need a procedure to compute Sat(Ψ) for Ψ given the sets Sat(Ψ′)for all state sub-formulas Ψ′ of Ψ:
LemmaI Sat(Φ1 ∧ Φ2) = Sat(Φ1) ∩ Sat(Φ2)
I Sat(¬Φ) = S \ Sat(Φ)
Sat(PJ(Φ)) = s | Ps(Paths(Φ)) ∈ J discussed on the next slide.
76 / 84
![Page 122: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/122.jpg)
PCTL Model Checking - Algorithm - Path Operator
LemmaI Next:
Ps(Paths(X Φ)) =
∑s′∈Sat(Φ)
P(s, s ′)
I Bounded Until:
Ps(Paths(Φ1 U ≤n Φ2)) =
Ps(Sat(Φ1) U ≤n Sat(Φ2))
I Unbounded Until:
Ps(Paths(Φ1 U Φ2)) =
Ps(Sat(Φ1) U Sat(Φ2))
As before:can be reduced to transient analysis and to unbounded reachability.
77 / 84
![Page 123: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/123.jpg)
PCTL Model Checking - Algorithm - Path Operator
LemmaI Next:
Ps(Paths(X Φ)) =∑
s′∈Sat(Φ)
P(s, s ′)
I Bounded Until:
Ps(Paths(Φ1 U ≤n Φ2)) = Ps(Sat(Φ1) U ≤n Sat(Φ2))
I Unbounded Until:
Ps(Paths(Φ1 U Φ2)) = Ps(Sat(Φ1) U Sat(Φ2))
As before:can be reduced to transient analysis and to unbounded reachability.
77 / 84
![Page 124: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/124.jpg)
PCTL Model Checking - Algorithm - Path Operator
LemmaI Next:
Ps(Paths(X Φ)) =∑
s′∈Sat(Φ)
P(s, s ′)
I Bounded Until:
Ps(Paths(Φ1 U ≤n Φ2)) = Ps(Sat(Φ1) U ≤n Sat(Φ2))
I Unbounded Until:
Ps(Paths(Φ1 U Φ2)) = Ps(Sat(Φ1) U Sat(Φ2))
As before:can be reduced to transient analysis and to unbounded reachability.
77 / 84
![Page 125: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/125.jpg)
PCTL Model Checking - Algorithm - Complexity
Precise algorithmComputation for every node in the parse tree and for every state:
I All node types except for path operator – trivial.I Next: Trivial.I Until: Solving equation systems can be done by polynomially
many elementary arithmetic operations.I Bounded until: Matrix vector multiplications can be done by
polynomial many elementary arithmetic operations as well.Overall complexity:Polynomial in |D|, linear in |Φ| and the maximum step bound n.
In practiceThe until and bounded until probabilities computed approximatively:
I rounding off probabilities in matrix-vector multiplication,I using approximative iterative methods without error guarantees.
78 / 84
![Page 126: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/126.jpg)
pLTL Model Checking Algorithm
79 / 84
![Page 127: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/127.jpg)
LTL Model Checking - Overview
Definition: LTL Model CheckingLet D = (S ,P, π0, L) be a DTMC, Ψ a LTL formula, s ∈ S , andp ∈ [0, 1]. The model checking problem is to decide whethers |= PDs (Paths(Ψ)) ≥ p.
TheoremThe LTL model checking can be decided in time O(|D| · 2|Ψ|).
Algorithm Outline
1. Construct from Ψ a deterministic Rabin automaton A recognizingwords satisfying Ψ, i.e. Paths(Ψ) := L(ω) ∈ (2Ap)∞ | ω |= Ψ
2. Construct a product DTMC D × A that “embeds” thedeterministic execution of A into the Markov chain.
3. Compute in D × A the probability of paths where A satisfies theacceptance condition.
80 / 84
![Page 128: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/128.jpg)
LTL Model Checking - Overview
Definition: LTL Model CheckingLet D = (S ,P, π0, L) be a DTMC, Ψ a LTL formula, s ∈ S , andp ∈ [0, 1]. The model checking problem is to decide whethers |= PDs (Paths(Ψ)) ≥ p.
TheoremThe LTL model checking can be decided in time O(|D| · 2|Ψ|).
Algorithm Outline1. Construct from Ψ a deterministic Rabin automaton A recognizing
words satisfying Ψ, i.e. Paths(Ψ) := L(ω) ∈ (2Ap)∞ | ω |= Ψ
2. Construct a product DTMC D × A that “embeds” thedeterministic execution of A into the Markov chain.
3. Compute in D × A the probability of paths where A satisfies theacceptance condition.
80 / 84
![Page 129: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/129.jpg)
LTL Model Checking - Overview
Definition: LTL Model CheckingLet D = (S ,P, π0, L) be a DTMC, Ψ a LTL formula, s ∈ S , andp ∈ [0, 1]. The model checking problem is to decide whethers |= PDs (Paths(Ψ)) ≥ p.
TheoremThe LTL model checking can be decided in time O(|D| · 2|Ψ|).
Algorithm Outline1. Construct from Ψ a deterministic Rabin automaton A recognizing
words satisfying Ψ, i.e. Paths(Ψ) := L(ω) ∈ (2Ap)∞ | ω |= Ψ2. Construct a product DTMC D × A that “embeds” the
deterministic execution of A into the Markov chain.
3. Compute in D × A the probability of paths where A satisfies theacceptance condition.
80 / 84
![Page 130: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/130.jpg)
LTL Model Checking - Overview
Definition: LTL Model CheckingLet D = (S ,P, π0, L) be a DTMC, Ψ a LTL formula, s ∈ S , andp ∈ [0, 1]. The model checking problem is to decide whethers |= PDs (Paths(Ψ)) ≥ p.
TheoremThe LTL model checking can be decided in time O(|D| · 2|Ψ|).
Algorithm Outline1. Construct from Ψ a deterministic Rabin automaton A recognizing
words satisfying Ψ, i.e. Paths(Ψ) := L(ω) ∈ (2Ap)∞ | ω |= Ψ2. Construct a product DTMC D × A that “embeds” the
deterministic execution of A into the Markov chain.3. Compute in D × A the probability of paths where A satisfies the
acceptance condition.
80 / 84
![Page 131: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/131.jpg)
LTL Model Checking - ω-Automata (1.)Deterministic Rabin automaton (DRA): (Q,Σ, δ, q0,Acc)
I a DFA with a different acceptance condition,I Acc = (Ei ,Fi ) | 1 ≤ i ≤ kI each accepting infinite path must visit for some i
I all states of Ei at most finitely often andI some state of Fi infinitely often.
ExampleGive some automata recognizing the language of formulas
I (a ∧ X b) ∨ aUc
I FGa
I GFa
Lemma (Vardi&Wolper’86, Safra’88)For any LTL formula Ψ there is a DRA A recognizing Paths(Ψ) with|A| ∈ 22O(|Ψ|) .
81 / 84
![Page 132: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/132.jpg)
LTL Model Checking - ω-Automata (1.)Deterministic Rabin automaton (DRA): (Q,Σ, δ, q0,Acc)
I a DFA with a different acceptance condition,I Acc = (Ei ,Fi ) | 1 ≤ i ≤ kI each accepting infinite path must visit for some i
I all states of Ei at most finitely often andI some state of Fi infinitely often.
ExampleGive some automata recognizing the language of formulas
I (a ∧ X b) ∨ aUc
I FGa
I GFa
Lemma (Vardi&Wolper’86, Safra’88)For any LTL formula Ψ there is a DRA A recognizing Paths(Ψ) with|A| ∈ 22O(|Ψ|) .
81 / 84
![Page 133: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/133.jpg)
LTL Model Checking - ω-Automata (1.)Deterministic Rabin automaton (DRA): (Q,Σ, δ, q0,Acc)
I a DFA with a different acceptance condition,I Acc = (Ei ,Fi ) | 1 ≤ i ≤ kI each accepting infinite path must visit for some i
I all states of Ei at most finitely often andI some state of Fi infinitely often.
ExampleGive some automata recognizing the language of formulas
I (a ∧ X b) ∨ aUc
I FGa
I GFa
Lemma (Vardi&Wolper’86, Safra’88)For any LTL formula Ψ there is a DRA A recognizing Paths(Ψ) with|A| ∈ 22O(|Ψ|) .
81 / 84
![Page 134: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/134.jpg)
LTL Model Checking - Product DTMC (2.)For a labelled DTMC D = (S ,P, π0, L) and a DRAA = (Q, 2Ap, δ, q0, (Ei ,Fi ) | 1 ≤ i ≤ k) we define
1. a DTMC D × A = (S × Q,P′, π′0):I P′((s, q), (s ′, q′)) = P(s, s ′) if δ(q, L(s ′)) = q′ and 0, otherwise;I π′0((s, qs)) = π0(s) if δ(q0, L(s)) = qs and 0, otherwise; and
2. (E ′i ,F ′i ) | 1 ≤ i ≤ k where for each i :I E ′i = (s, q) | q ∈ Ei , s ∈ S,I F ′i = (s, q) | q ∈ Fi , s ∈ S,
LemmaThe construction preserves probability of accepting as
PDs (Lang(A)) = PD×A(s,qs )(ω | ∃i : inf(ω) ∩ E ′i = ∅, inf(ω) ∩ F ′i 6= ∅)
where inf(ω) is the set of states visited in ω infinitely often.
Proof sketch.We have a one-to-one correspondence between executions of D andD × A (as A is deterministic), mapping Lang(A) to · · · , andpreserving probabilities.
82 / 84
![Page 135: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/135.jpg)
LTL Model Checking - Product DTMC (2.)For a labelled DTMC D = (S ,P, π0, L) and a DRAA = (Q, 2Ap, δ, q0, (Ei ,Fi ) | 1 ≤ i ≤ k) we define
1. a DTMC D × A = (S × Q,P′, π′0):I P′((s, q), (s ′, q′)) = P(s, s ′) if δ(q, L(s ′)) = q′ and 0, otherwise;I π′0((s, qs)) = π0(s) if δ(q0, L(s)) = qs and 0, otherwise; and
2. (E ′i ,F ′i ) | 1 ≤ i ≤ k where for each i :I E ′i = (s, q) | q ∈ Ei , s ∈ S,I F ′i = (s, q) | q ∈ Fi , s ∈ S,
LemmaThe construction preserves probability of accepting as
PDs (Lang(A)) = PD×A(s,qs )(ω | ∃i : inf(ω) ∩ E ′i = ∅, inf(ω) ∩ F ′i 6= ∅)
where inf(ω) is the set of states visited in ω infinitely often.
Proof sketch.We have a one-to-one correspondence between executions of D andD × A (as A is deterministic), mapping Lang(A) to · · · , andpreserving probabilities.
82 / 84
![Page 136: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/136.jpg)
LTL Model Checking - Product DTMC (2.)For a labelled DTMC D = (S ,P, π0, L) and a DRAA = (Q, 2Ap, δ, q0, (Ei ,Fi ) | 1 ≤ i ≤ k) we define
1. a DTMC D × A = (S × Q,P′, π′0):I P′((s, q), (s ′, q′)) = P(s, s ′) if δ(q, L(s ′)) = q′ and 0, otherwise;I π′0((s, qs)) = π0(s) if δ(q0, L(s)) = qs and 0, otherwise; and
2. (E ′i ,F ′i ) | 1 ≤ i ≤ k where for each i :I E ′i = (s, q) | q ∈ Ei , s ∈ S,I F ′i = (s, q) | q ∈ Fi , s ∈ S,
LemmaThe construction preserves probability of accepting as
PDs (Lang(A)) = PD×A(s,qs )(ω | ∃i : inf(ω) ∩ E ′i = ∅, inf(ω) ∩ F ′i 6= ∅)
where inf(ω) is the set of states visited in ω infinitely often.
Proof sketch.We have a one-to-one correspondence between executions of D andD × A (as A is deterministic), mapping Lang(A) to · · · , andpreserving probabilities.
82 / 84
![Page 137: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/137.jpg)
LTL Model Checking - Computing Acceptance Pr. (3.)
How to check the probability of accepting in D × A?
Identify the BSCCs (Cj)j of D × A that for some 1 ≤ i ≤ k ,1. contain no state from E ′i and2. contain some state from F ′i .
LemmaPD×A(s,qs )(ω | ∃i : inf(ω) ∩ E ′i = ∅, inf(ω) ∩ F ′i 6= ∅) = PD×A(s,qs )(♦
⋃j Cj).
Proof sketch.I Note that some BSCC of each finite DTMC is reached with
probability 1 (short paths with prob. bounded from below),I Rabin acceptance condition does not depend on any finite prefix
of the infinite word,I every state of a finite irreducible DTMC is visited infinitely often
with probability 1 regardless of the choice of initial state.CorollaryPDs (Lang(A)) = PD×A(s,qs )(♦
⋃j Cj).
83 / 84
![Page 138: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/138.jpg)
LTL Model Checking - Computing Acceptance Pr. (3.)
How to check the probability of accepting in D × A?Identify the BSCCs (Cj)j of D × A that for some 1 ≤ i ≤ k ,
1. contain no state from E ′i and2. contain some state from F ′i .
LemmaPD×A(s,qs )(ω | ∃i : inf(ω) ∩ E ′i = ∅, inf(ω) ∩ F ′i 6= ∅) = PD×A(s,qs )(♦
⋃j Cj).
Proof sketch.I Note that some BSCC of each finite DTMC is reached with
probability 1 (short paths with prob. bounded from below),I Rabin acceptance condition does not depend on any finite prefix
of the infinite word,I every state of a finite irreducible DTMC is visited infinitely often
with probability 1 regardless of the choice of initial state.CorollaryPDs (Lang(A)) = PD×A(s,qs )(♦
⋃j Cj).
83 / 84
![Page 139: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/139.jpg)
LTL Model Checking - Computing Acceptance Pr. (3.)
How to check the probability of accepting in D × A?Identify the BSCCs (Cj)j of D × A that for some 1 ≤ i ≤ k ,
1. contain no state from E ′i and2. contain some state from F ′i .
LemmaPD×A(s,qs )(ω | ∃i : inf(ω) ∩ E ′i = ∅, inf(ω) ∩ F ′i 6= ∅) = PD×A(s,qs )(♦
⋃j Cj).
Proof sketch.I Note that some BSCC of each finite DTMC is reached with
probability 1 (short paths with prob. bounded from below),I Rabin acceptance condition does not depend on any finite prefix
of the infinite word,I every state of a finite irreducible DTMC is visited infinitely often
with probability 1 regardless of the choice of initial state.
CorollaryPDs (Lang(A)) = PD×A(s,qs )(♦
⋃j Cj).
83 / 84
![Page 140: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/140.jpg)
LTL Model Checking - Computing Acceptance Pr. (3.)
How to check the probability of accepting in D × A?Identify the BSCCs (Cj)j of D × A that for some 1 ≤ i ≤ k ,
1. contain no state from E ′i and2. contain some state from F ′i .
LemmaPD×A(s,qs )(ω | ∃i : inf(ω) ∩ E ′i = ∅, inf(ω) ∩ F ′i 6= ∅) = PD×A(s,qs )(♦
⋃j Cj).
Proof sketch.I Note that some BSCC of each finite DTMC is reached with
probability 1 (short paths with prob. bounded from below),I Rabin acceptance condition does not depend on any finite prefix
of the infinite word,I every state of a finite irreducible DTMC is visited infinitely often
with probability 1 regardless of the choice of initial state.CorollaryPDs (Lang(A)) = PD×A(s,qs )(♦
⋃j Cj).
83 / 84
![Page 141: Quantitative Verification - Chapter 3: Markov chainskretinsk/teaching/qv/3.pdf · Quantitative Verification Chapter 3: Markov chains Jan Kˇret´ınsky´ Technical University of](https://reader033.fdocuments.net/reader033/viewer/2022041711/5e482e2d076ceb41f054ed0c/html5/thumbnails/141.jpg)
LTL Model Checking - Algorithm - Complexity
Doubly exponential in Ψ and polynomial in D(for the algorithm presented here):
1. |A| and hence also |D × A| is of size 22O(|Ψ|)
2. BSCC computation: Tarjan algorithm - linear in |D × A|(number of states + transitions)
3. Unbounded reachability: system of linear equations (≤ |D×A|):I exact solution: ≈ cubic in the size of the systemI approximative solution: efficient in practice
84 / 84