Performance Model CheckingScenario-Aware Dataflow
Bart Theelen, Marc Geilen, Jeroen Voeten
2 Overview
• Dataflow Formalisms• Timed Probabilistic Systems• Performance Model Checking• Experimental Results• Conclusions & Outlook
Dataflow Formalisms
• Example digital signal processing areas
3
Streaming Multi-Media Loop-Control in Mechatronics
Req. Reorder AntialiasHybrid Synth.
Freq. Inv.
Subb. Inv.
Req. Reorder
Stereo
AntialiasHybrid Synth.
Freq. Inv.
Subb. Inv.
WriteBackHuffman
Frame Detect
Block Detect
Block Detect
Dataflow formalisms describe task graphswhere potential parallelism is made explicit
MP3 Decoder
Dataflow Formalisms: Expressivity vs Analyzability
4
analyzabilityimplementation efficiency
low
high
expressiveness and succinctness
HSDF / MG
SDF / WMG
CG
CSDF
BDF
KPN
FSM-SADF / HDF
VRDF
SADF
DDF
RPN
VPDF
PSDF
expressivenesshierarchy
HSDF / MG
SDF / WMG
CG
CSDF
BDF
KPN
FSM-SADF / HDF
VRDF
SADF
DDF
RPN
VPDF
PSDF
Stuijk, et al. Scenario-Aware Dataflow: Modeling, Analysis and Implementation of Dynamic Applications. SAMOS’11
Synchronous Dataflow(Weighted Marked Graphs)
Kahn ProcessNetworks
Scenario-AwareDataflow
Scenario-Aware Dataflow (SADF)
• Scenario = operation modes with similar resource usage• Detectors control processes by sending scenario-valued
tokens• Detectors contain automata to capture scenario
occurrences• Real-life: data-dependent control behaviour (normal state
machine)• Modelling worst/best-case only: non-deterministic state
machine• Modelling worst/best-case & average-case: Markov chain
5
d
a
11
1
1
d
1
1
11
b
1
c
1
1
d
e
31
1
c
IDCTVLD
MCRCFD
kernel
data channel
detector
Rate I P0 Px
a 0 0 1
b 0 0 x
c 99 1 x
d 1 0 1
e 99 0 x
x = {30, 40, 50 ,60, 70, 80, 99}
P0
I
P99
P0
I
P99pI→P99
pI→P0
MPEG-4 Decoder
control channel
rate
tokens
• Processes run in parallel according to ‘extended actor semantics’1. Determine scenario depending on
1. Kernels & Detectors: scenario-valued control tokens2. Detectors: next state of Markov chain
2. Wait until sufficient tokens available3. Perform the actual task (sample from discrete time
distribution)4. Produce and consume tokens
d
a
11
1
1
d
1
1
11
b
1
c
1
1
d
e
31
1
c
IDCTVLD
MCRCFD
Scenario-Aware Dataflow (SADF)6
Rate I P0 Px
a 0 0 1
b 0 0 x
c 99 1 x
d 1 0 1
e 99 0 x
x = {30, 40, 50 ,60, 70, 80, 99}
MPEG-4 Decoder
Timed Probabilistic Systems(TPS)
• Compositional semantic model with• guarded (interactive) action transitions• probabilistic transitions• deterministic time transitions
• Alternates action/time transitions with probabilistic fan-out
• Pattern for generic discrete execution time distributions
time advances exactly ti time units with probability pi for i=1,…,n
7
a
p
t
t1
τ
p1
pn 1
1
tn
Illustrative Example8
Rate ς1 ς2
a 2 0b 1 0
Process ScenarioExecution
TimeProbabilit
y
A ς1
19 8/1151 1/1157 2/11
B
ς1
5 5/1917 12/1947 2/19
ς2
13 1/1531 13/1563 1/15
D
ς1
3 4/1413 1/1429 9/14
ς2
7 3/419 1/4
a
1
2
11
11
b
11
BA
D
a
b
S1
1
19
control (ς1)
S2
control (ς2)
1 S3
S4
start
1end
1
start
S6
S5
S7
8/11
1/112/11
51
57
S8
1
1
1
end
1
TPS for Kernel A
S1
3
S2
detect
start
S5
S4
S6
4/14
1/149/14
13
29S9
1
1
1
end
1
7
S3start S8
S73/4
1/4
19 S10
1
1
2/3 (ς1)
1/3 (ς2)
3
S12
start
S14
S13
S15
4/14
1/149/14
13
29S16
1
1
1
S11detect1 (ς1)
end
1
1
end
TPS for Detector D
ς1 ς2
1/3
1
2/3
Semantic Properties
• Model checking based on (relevant) state-space• Exploit semantic properties to limit state-space explosion
• SADF satisfies various semantic properties• Time additivity, action persistency, action urgency, action
determinacy
• Only non-determinism between actions as a result of concurrency• Policy for resolving non-determinism does not effect net behaviour
9
S1
a1 a2
S2 S3 S4
1 p2p1
S5 S6
a2
p1p2
a1 a1
11
policy for resolving non-determinism may however effect performance result
Performance Model Checking
Direct computation of quantitative resultsbased on model checking techniques
• Broad variety of performance metrics• Mostly complex reward-based properties • Mostly time-related properties
10
Metric TypeProbabilities
RelevantScope
Max Buffer Occupancy Worst Case No All States
Min/Max Response Delay Best/Worst Case No Transient
Min/Max Inter-Firing Delay Best/Worst Case No All States
Response Deadline Miss Probability
Probabilistic Reachability Yes Transient
Expected Response Delay Expected Reachability Yes Transient
Throughput Event Rate Yes Steady State
Periodic Deadline Miss ProbabilitySample Average /
Expected ReachabilityYes Steady State
Average Inter-Firing Latency Sample Average Yes Steady State
Variance in Inter-Firing Latency Sample Variance Yes Steady State
Average Buffer Occupancy Time-Weighted Average Yes Steady State
Variance in Buffer Occupancy Time-Weighted Variance Yes Steady State
Policy for resolving non-determinism only
affects Max Buffer Occupancy
Model Checking Strategy - Theory11
TPS per SADF Process
Discrete Markov Chain
Move Transition Labels into States
|S|
Deterministic TPS of Complete SADF Model
Resolve Non-Determinism
|S’|
TPS of Complete SADF Model
Parallel Composition
|S|
p2
S2
a
p1S1
S3
S1, -
S2, a
S3, a
p1
p2
Information on occurrence of actions and time available
through reward functions on states only
≤|S|
>|S’|
Reduced Discrete Markov Reward Model
Remove Irrelevant States
|Sc|<<|S|
Performance Number
Compute Equilibrium
Throughput
• MPEG-4 throughput = average number of frames per second
= average number of RC firings per time unit
• Let {Xi | i ≥ 1} denote Markov chain with state space S• Define reward c to identify firing completion action of RC• If c(s) = 1 for state s, it is relevant, otherwise it is irrelevant
12
n
2ii1i
n
2ii
nt)c(X)Δ(X
)c(Xlim
t
Firings RC#lim
c
Time0
1
t1 t4t3t2
Δ1 Δ2 Δ3 Δ4
Δ is temporal reward function denoting amount of time elapsed since previous RC
firing
Metric-Dependent State-Space Reduction
• If relevant positive recurrent state for ergodic Markov chain exists, then• Reduction yields ergodic Markov chain• Reduction preserves performance properties
13
C
BA D
E
F
1
G
1/2
1/4
1
1/3
1/6
1/2
1/3
1/2B D
E
G
1/2
1 5/12
1/12
1/6
c cvu,Su Sv Ωρ
ρρcu
n
2ii1i
n
2ii
nt Δπ1
)c(X)Δ(X
)c(Xlim
t
Firings RC#lim
P
Transformed TPS per SADF Process
Model Checking Strategy - Practice14
TPS per SADF Process
Discrete Markov Chain
Move Transition Labels into States
|S|
Deterministic TPS of Complete SADF Model
Resolve Non-Determinism
|S’|
TPS of Complete SADF Model
Parallel Composition
|S|
Reduced Discrete Markov Reward Model
Remove Irrelevant States
|Sc|
Performance Number
Compute Equilibrium
Transformed TPS of Complete SADF Model |S’’|
Discrete Markov Chain |S|
Parallel composition with on-the-fly reduction
and resolving non-determinism
Experimental Results
- requires more than 1.5GB of memory¤ takes more than 6 hours
Reduction after resolving non-determinism
15
Case Study |S’’| |S|
MPEG-4 AVC 185 183
Channel Equalizer
2185 296
MPEG-4 SP (PD = 1)
- 38440
MPEG-4 SP (PD = 2)
- 483400
MPEG-4 SP (PD = 3)
- -
MP3 (PD = 1) - -
MP3 (PD = 2) - -
MP3 (PD = 3) - -
MP3 (4 ≤ PD ≤ 9) - -
|Sc| Factor
18 10.2
8 37
9 4271.1
576 839.2
8253 -
5 -
15 -
15 -
¤ -
Time [s]
Memory [MB]
≤ 0.001 0.272
0.012 0.672
0.8 7.9
40.7 16.3
906.9 94
26.8 64.6
624.6 165
20356 275.5
> 6h ¤
d
a
11
1
1
d
1
1
11
b
1
c
1
1
d
e
PD1
1
c
IDCTVLD
MCRCFD
Statistical Model Checking as Alternative
• Statistical model checking supported by modelling SADF in POOSL• POOSL is much more expressive than SADF but also has TPS
semantics• Compositional estimation of confidence intervals for long-run
averages
16
Case Study Throughput Time [s]
MPEG-4 AVC 131.027 ≤ 0.001
Channel Equalizer
0.162443·10-
3
0.012
MPEG-4 SP (PD = 1)
0.745268·10-
3
0.8
MPEG-4 SP (PD = 2)
1.05388·10-3 40.7
MPEG-4 SP (PD = 3)
1.06378·10-3 906.9
MP3 (PD = 1) 2.33449·10-7 26.8
MP3 (PD = 2) 2.68096·10-7 624.6
MP3 (PD = 3) 2.68096·10-7 20356
MP3 (PD = 9) ¤ > 6h
95% Confidence Interval Time [s]
[131.027, 131.027] 0.14
[0.162443·10-3, 0. 162443·10-3]
0.75
[0.744976·10-3, 0.747931·10-
3]7.4
[1.03899·10-3, 1.08953·10-3] 6.86
[1.04035·10-3, 1.06344·10-3] 6.85
[2.33340·10-7, 2.33383·10-7] 32.8
[2.68096·10-7, 2.68096·10-7] 31.5
[2.68096·10-7, 2.68096·10-7] 32.1
[2.68096·10-7, 2.68096·10-7] 30.6
Conclusions & Outlook
• Performance model checking approach for SADF• Compositional TPS semantics with discrete time distributions• Exploit semantic properties• Removal of metric-dependent irrelevant states• On-the-fly construction of relevant state-space• Broad variety of pre-defined performance metrics
• All expressible as temporal reward formula• Statistical model-checking for long-run averages as alternative
• Increase flexibility to allow computing user-defined metrics• Lift Markov chain reduction to bisimulation reduction on TPS• Support temporal rewards as property specification language
Could contemporary quantitative model checkers supporting Probabilistic Timed Automata be a suitable alternative?
17
Top Related