Performance Model CheckingScenario-Aware Dataflow

Bart Theelen, Marc Geilen, Jeroen Voeten

2 Overview

• Dataflow Formalisms• Timed Probabilistic Systems• Performance Model Checking• Experimental Results• Conclusions & Outlook

Dataflow Formalisms

• Example digital signal processing areas

3

Streaming Multi-Media Loop-Control in Mechatronics

Req. Reorder AntialiasHybrid Synth.

Freq. Inv.

Subb. Inv.

Req. Reorder

Stereo

AntialiasHybrid Synth.

Freq. Inv.

Subb. Inv.

WriteBackHuffman

Frame Detect

Block Detect

Block Detect

Dataflow formalisms describe task graphswhere potential parallelism is made explicit

MP3 Decoder

Dataflow Formalisms: Expressivity vs Analyzability

4

analyzabilityimplementation efficiency

low

high

expressiveness and succinctness

HSDF / MG

SDF / WMG

CG

CSDF

BDF

KPN

FSM-SADF / HDF

VRDF

SADF

DDF

RPN

VPDF

PSDF

expressivenesshierarchy

HSDF / MG

SDF / WMG

CG

CSDF

BDF

KPN

FSM-SADF / HDF

VRDF

SADF

DDF

RPN

VPDF

PSDF

Stuijk, et al. Scenario-Aware Dataflow: Modeling, Analysis and Implementation of Dynamic Applications. SAMOS’11

Synchronous Dataflow(Weighted Marked Graphs)

Kahn ProcessNetworks

Scenario-AwareDataflow

Scenario-Aware Dataflow (SADF)

• Scenario = operation modes with similar resource usage• Detectors control processes by sending scenario-valued

tokens• Detectors contain automata to capture scenario

occurrences• Real-life: data-dependent control behaviour (normal state

machine)• Modelling worst/best-case only: non-deterministic state

machine• Modelling worst/best-case & average-case: Markov chain

5

d

a

11

1

1

d

1

1

11

b

1

c

1

1

d

e

31

1

c

IDCTVLD

MCRCFD

kernel

data channel

detector

Rate I P0 Px

a 0 0 1

b 0 0 x

c 99 1 x

d 1 0 1

e 99 0 x

x = {30, 40, 50 ,60, 70, 80, 99}

P0

I

P99

P0

I

P99pI→P99

pI→P0

MPEG-4 Decoder

control channel

rate

tokens

• Processes run in parallel according to ‘extended actor semantics’1. Determine scenario depending on

1. Kernels & Detectors: scenario-valued control tokens2. Detectors: next state of Markov chain

2. Wait until sufficient tokens available3. Perform the actual task (sample from discrete time

distribution)4. Produce and consume tokens

d

a

11

1

1

d

1

1

11

b

1

c

1

1

d

e

31

1

c

IDCTVLD

MCRCFD

Scenario-Aware Dataflow (SADF)6

Rate I P0 Px

a 0 0 1

b 0 0 x

c 99 1 x

d 1 0 1

e 99 0 x

x = {30, 40, 50 ,60, 70, 80, 99}

MPEG-4 Decoder

Timed Probabilistic Systems(TPS)

• Compositional semantic model with• guarded (interactive) action transitions• probabilistic transitions• deterministic time transitions

• Alternates action/time transitions with probabilistic fan-out

• Pattern for generic discrete execution time distributions

time advances exactly ti time units with probability pi for i=1,…,n

7

a

p

t

t1

τ

p1

pn 1

1

tn

Illustrative Example8

Rate ς1 ς2

a 2 0b 1 0

Process ScenarioExecution

TimeProbabilit

y

A ς1

19 8/1151 1/1157 2/11

B

ς1

5 5/1917 12/1947 2/19

ς2

13 1/1531 13/1563 1/15

D

ς1

3 4/1413 1/1429 9/14

ς2

7 3/419 1/4

a

1

2

11

11

b

11

BA

D

a

b

S1

1

19

control (ς1)

S2

control (ς2)

1 S3

S4

start

1end

1

start

S6

S5

S7

8/11

1/112/11

51

57

S8

1

1

1

end

1

TPS for Kernel A

S1

3

S2

detect

start

S5

S4

S6

4/14

1/149/14

13

29S9

1

1

1

end

1

7

S3start S8

S73/4

1/4

19 S10

1

1

2/3 (ς1)

1/3 (ς2)

3

S12

start

S14

S13

S15

4/14

1/149/14

13

29S16

1

1

1

S11detect1 (ς1)

end

1

1

end

TPS for Detector D

ς1 ς2

1/3

1

2/3

Semantic Properties

• Model checking based on (relevant) state-space• Exploit semantic properties to limit state-space explosion

• SADF satisfies various semantic properties• Time additivity, action persistency, action urgency, action

determinacy

• Only non-determinism between actions as a result of concurrency• Policy for resolving non-determinism does not effect net behaviour

9

S1

a1 a2

S2 S3 S4

1 p2p1

S5 S6

a2

p1p2

a1 a1

11

policy for resolving non-determinism may however effect performance result

Performance Model Checking

Direct computation of quantitative resultsbased on model checking techniques

• Broad variety of performance metrics• Mostly complex reward-based properties • Mostly time-related properties

10

Metric TypeProbabilities

RelevantScope

Max Buffer Occupancy Worst Case No All States

Min/Max Response Delay Best/Worst Case No Transient

Min/Max Inter-Firing Delay Best/Worst Case No All States

Response Deadline Miss Probability

Probabilistic Reachability Yes Transient

Expected Response Delay Expected Reachability Yes Transient

Throughput Event Rate Yes Steady State

Periodic Deadline Miss ProbabilitySample Average /

Expected ReachabilityYes Steady State

Average Inter-Firing Latency Sample Average Yes Steady State

Variance in Inter-Firing Latency Sample Variance Yes Steady State

Average Buffer Occupancy Time-Weighted Average Yes Steady State

Variance in Buffer Occupancy Time-Weighted Variance Yes Steady State

Policy for resolving non-determinism only

affects Max Buffer Occupancy

Model Checking Strategy - Theory11

TPS per SADF Process

Discrete Markov Chain

Move Transition Labels into States

|S|

Deterministic TPS of Complete SADF Model

Resolve Non-Determinism

|S’|

TPS of Complete SADF Model

Parallel Composition

|S|

p2

S2

a

p1S1

S3

S1, -

S2, a

S3, a

p1

p2

Information on occurrence of actions and time available

through reward functions on states only

≤|S|

>|S’|

Reduced Discrete Markov Reward Model

Remove Irrelevant States

|Sc|<<|S|

Performance Number

Compute Equilibrium

Throughput

• MPEG-4 throughput = average number of frames per second

= average number of RC firings per time unit

• Let {Xi | i ≥ 1} denote Markov chain with state space S• Define reward c to identify firing completion action of RC• If c(s) = 1 for state s, it is relevant, otherwise it is irrelevant

12

n

2ii1i

n

2ii

nt)c(X)Δ(X

)c(Xlim

t

Firings RC#lim

c

Time0

1

t1 t4t3t2

Δ1 Δ2 Δ3 Δ4

Δ is temporal reward function denoting amount of time elapsed since previous RC

firing

Metric-Dependent State-Space Reduction

• If relevant positive recurrent state for ergodic Markov chain exists, then• Reduction yields ergodic Markov chain• Reduction preserves performance properties

13

C

BA D

E

F

1

G

1/2

1/4

1

1/3

1/6

1/2

1/3

1/2B D

E

G

1/2

1 5/12

1/12

1/6

c cvu,Su Sv Ωρ

ρρcu

n

2ii1i

n

2ii

nt Δπ1

)c(X)Δ(X

)c(Xlim

t

Firings RC#lim

P

Transformed TPS per SADF Process

Model Checking Strategy - Practice14

TPS per SADF Process

Discrete Markov Chain

Move Transition Labels into States

|S|

Deterministic TPS of Complete SADF Model

Resolve Non-Determinism

|S’|

TPS of Complete SADF Model

Parallel Composition

|S|

Reduced Discrete Markov Reward Model

Remove Irrelevant States

|Sc|

Performance Number

Compute Equilibrium

Transformed TPS of Complete SADF Model |S’’|

Discrete Markov Chain |S|

Parallel composition with on-the-fly reduction

and resolving non-determinism

Experimental Results

- requires more than 1.5GB of memory¤ takes more than 6 hours

Reduction after resolving non-determinism

15

Case Study |S’’| |S|

MPEG-4 AVC 185 183

Channel Equalizer

2185 296

MPEG-4 SP (PD = 1)

- 38440

MPEG-4 SP (PD = 2)

- 483400

MPEG-4 SP (PD = 3)

- -

MP3 (PD = 1) - -

MP3 (PD = 2) - -

MP3 (PD = 3) - -

MP3 (4 ≤ PD ≤ 9) - -

|Sc| Factor

18 10.2

8 37

9 4271.1

576 839.2

8253 -

5 -

15 -

15 -

¤ -

Time [s]

Memory [MB]

≤ 0.001 0.272

0.012 0.672

0.8 7.9

40.7 16.3

906.9 94

26.8 64.6

624.6 165

20356 275.5

> 6h ¤

d

a

11

1

1

d

1

1

11

b

1

c

1

1

d

e

PD1

1

c

IDCTVLD

MCRCFD

Statistical Model Checking as Alternative

• Statistical model checking supported by modelling SADF in POOSL• POOSL is much more expressive than SADF but also has TPS

semantics• Compositional estimation of confidence intervals for long-run

averages

16

Case Study Throughput Time [s]

MPEG-4 AVC 131.027 ≤ 0.001

Channel Equalizer

0.162443·10-

3

0.012

MPEG-4 SP (PD = 1)

0.745268·10-

3

0.8

MPEG-4 SP (PD = 2)

1.05388·10-3 40.7

MPEG-4 SP (PD = 3)

1.06378·10-3 906.9

MP3 (PD = 1) 2.33449·10-7 26.8

MP3 (PD = 2) 2.68096·10-7 624.6

MP3 (PD = 3) 2.68096·10-7 20356

MP3 (PD = 9) ¤ > 6h

95% Confidence Interval Time [s]

[131.027, 131.027] 0.14

[0.162443·10-3, 0. 162443·10-3]

0.75

[0.744976·10-3, 0.747931·10-

3]7.4

[1.03899·10-3, 1.08953·10-3] 6.86

[1.04035·10-3, 1.06344·10-3] 6.85

[2.33340·10-7, 2.33383·10-7] 32.8

[2.68096·10-7, 2.68096·10-7] 31.5

[2.68096·10-7, 2.68096·10-7] 32.1

[2.68096·10-7, 2.68096·10-7] 30.6

Conclusions & Outlook

• Performance model checking approach for SADF• Compositional TPS semantics with discrete time distributions• Exploit semantic properties• Removal of metric-dependent irrelevant states• On-the-fly construction of relevant state-space• Broad variety of pre-defined performance metrics

• All expressible as temporal reward formula• Statistical model-checking for long-run averages as alternative

• Increase flexibility to allow computing user-defined metrics• Lift Markov chain reduction to bisimulation reduction on TPS• Support temporal rewards as property specification language

Could contemporary quantitative model checkers supporting Probabilistic Timed Automata be a suitable alternative?

17