Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers
-
Upload
lionel-briand -
Category
Software
-
view
357 -
download
1
Transcript of Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers
![Page 1: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/1.jpg)
.lusoftware verification & validationVVS
Effective Test Suites for !Mixed Discrete-Continuous
Stateflow Controllers Reza Matinnejad Shiva Nejati Lionel Briand SnT Center, University of Luxembourg
Thomas Bruckmann Delphi Automotive Systems, Luxembourg
![Page 2: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/2.jpg)
Cyber Physical Systems (CPSs) Combination of computations (algorithms) and physical dynamics (differential equations)
2
Physical world Computation
![Page 3: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/3.jpg)
Testing (Typical) Software
3
X = 10, Y = 30
Z = 20
Algorithms
Fail Pass Z = 10
![Page 4: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/4.jpg)
Testing (CPS) Software
4
Algorithms + Differential Equations
Fail Z = 20
X = 10, Y = 30 S1(t) S2(t)
S3(t) Pass
Z = 20 S3(t)
S1t
S2t
S3t
S3t
![Page 5: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/5.jpg)
Software Testing Challenges (CPS)
• Mixed discrete-continuous behavior (combination of algorithms and continuous dynamics)
• Inputs/outputs are signals (functions over time)
• Simulation is inexpensive but not yet systematically automated
• Partial test oracles
5
![Page 6: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/6.jpg)
Generating effective test suites for Software used in !
Cyber-Physical Systems
6
Our Goal
![Page 7: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/7.jpg)
Simulink/Stateflow
• A data flow-driven block diagram language
• Is widely used to develop Cyber Physical Systems
• Is executable
7
![Page 8: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/8.jpg)
Stateflow
• A Statechart dialect integrated into Simulink
• Captures the state-based behavior of CPS software
• Has mixed discrete-continuous behavior
8
![Page 9: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/9.jpg)
Generating effective test suites for mixed discrete-continuous
Stateflow controllers
9
Our Goal
![Page 10: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/10.jpg)
Discrete Behavior What we typically think of software models
10
On
Off
On
Off
Speed < 10 Speed > 10
![Page 11: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/11.jpg)
Discrete-Continuous Behavior What software models are actually being built using Stateflow
11
On
Off
CtrlSig
On
Off
Speed < 10 Speed > 10
tCtrlSig
tCtrlSig
![Page 12: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/12.jpg)
Generating effective test suites for mixed discrete-continuous
Stateflow controllers
12
Our Goal
![Page 13: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/13.jpg)
Test Suite Effectiveness (1) • Test suite size should be small because
• Test oracles cannot be fully automated
• Output signals need to be inspected by engineers
13
ModelSimulation
InputSignals
OutputSignal(s)
S3t
S2t
S1t
S3t
S2t
S1t
Test Case 1
Test Case 2
![Page 14: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/14.jpg)
Test Suite Effectiveness (2) • Test suites should have a high fault revealing power
• Small deviations in outputs may not be recognized/important
• Test inputs that drastically impact the output signal shape are likely to have a higher fault revealing power
14
Test Output 1
TimeTime
CtrlSig
Faulty Model OutputCorrect Model Output
Test Output 2
![Page 15: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/15.jpg)
Test Generation Algorithms!!
15
Our Approach
![Page 16: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/16.jpg)
Test Generation Algorithms • Input-based Test Generation:
• Input Diversity Algorithm
• Coverage-based Test Generation:
• State Coverage Algorithm
• Transition Coverage Algorithm
• Output-based Test Generation:
• Output Diversity Algorithm
• Failure-based Algorithm
16
![Page 17: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/17.jpg)
Input Diversity • Maximizing distances among input signals
17
Test Case 1
Test Case 2
Input Signal 1 Input Signal 2
S1t
S1t
S2t
S2t
![Page 18: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/18.jpg)
Distance Between Signals
18
Time
Signal
![Page 19: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/19.jpg)
Test Generation Algorithms • Input-based Test Generation:
• Input Diversity Algorithm
• Coverage-based Test Generation:
• State Coverage Algorithm
• Transition Coverage Algorithm
• Output-based Test Generation:
• Output Diversity Algorithm
• Failure-based Algorithm
19
![Page 20: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/20.jpg)
Structural Coverage
• Maximizing the number of states/transitions covered
20
State Coverage Transition Coverage
1
4
2
3
1
4
2
3
![Page 21: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/21.jpg)
Test Generation Algorithms • Input-based Test Generation:
• Input Diversity Algorithm
• Coverage-based Test Generation:
• State Coverage Algorithm
• Transition Coverage Algorithm
• Output-based Test Generation:
• Output Diversity Algorithm
• Failure-based Algorithm
21
![Page 22: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/22.jpg)
Output Diversity • Maximizing distances among output signals
22
Test Case 1
Test Case 2
Output Signal
S3t
S3t
![Page 23: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/23.jpg)
Failure-based Test Generation
23
Instability Discontinuity
0.0 1.0 2.0-1.0
-0.5
0.0
0.5
1.0
Time
Ctr
lSig
Output
• Maximizing the likelihood of presence of specific failure patterns in output signals
0.0 1.0 2.0Time
0.0
0.25
0.50
0.75
1.0
Ctr
lSig
Output
![Page 24: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/24.jpg)
We developed our failure-based test generation algorithm using!
Meta-Heuristic Search
24
![Page 25: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/25.jpg)
The Alternative Choice
25
Our ApproachExisting WorkTechnique
ModelChecking
- Require precisely definedoracles (user-specified assertions)
- Have been largely appliedto time-discrete models
- State-explosion problem!
- No need for automated test oracles
- Applicable to time-continuousand non-linear models
- Our algorithms are black-boxrandomized search: - non-memory intensive - can be parallelized
![Page 26: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/26.jpg)
26
Failure-based Test Generation using Meta-Heuristic Search
Input Signals
Slightly Modifying Each Input Signal
Fitness Functions Capturing the Likelihood
of Presence of Failure Patterns in the Output Signals
Repeat
Until maximum resources spent
S Initial Candidate Solution
Search Procedure
R Tweak (S)
if Fitness (R) > Fitness (S)
S R
Return S
![Page 27: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/27.jpg)
Output Stability !Fitness Function
• Sum of the differences of signal values for consecutive simulation steps
27
stability(sgo
) =kP
i=1|sg
o
(i ·�t)� sgo
((i� 1) ·�t)|
0.0 1.0 2.0-1.0
-0.5
0.0
0.5
1.0
Time
Ctr
lSig
Output
![Page 28: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/28.jpg)
Output Continuity !Fitness Function
28
• Maximum of the minimum left or right derivatives for all the simulation steps
0.0 1.0 2.0Time
0.0
0.25
0.50
0.75
1.0
Ctr
lSig
Output
continuity(sg
o
) =
K�1max
i=1(min(|LeftDer(sg
o
, i)|, |RightDer(sgo
, i)|))
![Page 29: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/29.jpg)
Comparing the!Test Generation Algorithms!
!
29
Evaluation
![Page 30: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/30.jpg)
Research Questions
• RQ1 (Fault Revealing Ability)
• RQ2 (Fault Revealing Subsumption)
• RQ3 (Test Suite Size)
30
![Page 31: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/31.jpg)
Experiment Setup • Three Stateflow models: two industrial and one publicly
available case study
31
75 (faulty models) * 100 (algorithm runs) *6 (generation algorithms) * 5 (different test suite sizes) =
225,000 test suites (in total)
Test Suite(size=3,5, 10,25,50)
{1.Fault
Seeding2.Generation
AlgorithmSF FaultySF
{75 75
![Page 32: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/32.jpg)
Research Question 1!Fault Revealing Ability
How does the fault revealing ability of our proposed test generation algorithms
compare with one another?
32
![Page 33: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/33.jpg)
1.0
0.0
0.5
Input Diversity
OutputDiversity
FaultRevealing
Rate
RQ1: Fault Revealing Ability
33
1. Output-based and coverage-based algorithms outperformed the input diversity algorithm
2. Output-based algorithms outperformed the coverage-based algorithms
3. Overall, output stability algorithm performed the best
![Page 34: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/34.jpg)
Research Question 2!Fault Revealing Subsumption
Is any of our generation algorithms subsumed by other algorithms?
34
![Page 35: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/35.jpg)
RQ2: Fault Revealing Subsumption
35
• For each of the 75 faulty models, we identified the best generation algorithm(s) for different test suite sizes (5, 10, 25, and 50)
Fault 1State Coverage
Transition Coverage
Output Diversity
Output Stability
Output Continuity
Fault 2 Fault 3 Fault 4
![Page 36: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/36.jpg)
RQ2: Fault Revealing Subsumption (2)
36
1. The coverage-based algorithms found the least number of faults
2. Coverage-based algorithms are subsumed by output diversity algorithm when the test suite size increases (size = 25 , 50)
![Page 37: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/37.jpg)
Research Question 3!Test Suite Size
What is the impact of the size of test suites generated by our generation algorithms on
their fault revealing ability?
37
![Page 38: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/38.jpg)
RQ3: Test Suite Size
38
1. The fault revealing rates for output stability/continuity is very high for small test suites(size = 3,5) for Instability/Discontinuity failures
2. For Other failures, the ability of output diversity in revealing failures rapidly increases as the test suite size increases
DiscontinuityInstability Others
0.0
0.5
1.0
3 5 10 25 50
Test Suite Size
Faul
t Rev
ealin
g Ra
te M
ean
3 5 10 25 50 3 5 10 25 50
Output StabilityOuput Continuity State Coverage
Transition CoverageOutput Diversity
![Page 39: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/39.jpg)
Lessons Learned
39
![Page 40: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/40.jpg)
Lesson 1!Coverage-based algorithms are less
effective than output-based algorithms • The test cases resulting from state/transition coverage
algorithms cover the faulty parts of the models
• 97% state coverage and 81% transition coverage
• Cover faulty parts for 73 (out of 75) fault-seeded models
• However, they fail to generate output signals that are sufficiently distinct from the oracle signal, hence yielding a low fault revealing rate
40
![Page 41: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/41.jpg)
Lesson 2!Combining Output-based Algorithms
41
• We suggest to divide the test suite size budget between output-based algorithms:
Output Continuity Output Stability Output Diversity
![Page 42: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/42.jpg)
CoCoTest
42
![Page 43: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/43.jpg)
.lusoftware verification & validationVVS
Effective Test Suites for !Mixed Discrete-Continuous
Stateflow Controllers Reza Matinnejad ([email protected]) Shiva Nejati Lionel Briand SnT Center, University of Luxembourg
Thomas Bruckmann Delphi Automotive Systems, Luxembourg
![Page 44: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/44.jpg)
Lesson 1!Combing Output-based Algorithms
• We suggest to divide the test suite size budget between output stability, output continuity, and output diversity:
1. Allocate a small part of the test budget to output continuity
2. Share the rest of the budget between output stability and output diversity, by giving output diversity a higher share
44
![Page 45: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/45.jpg)
Input / Output Vectors
45
0 5 10
50
150
250
Fuel
Lev
el S
enso
r
Fuel
Lev
el0 5 10
100.0
91.43
84.43
75.6270.01
66.1961.21
56.6654.3252.81
50
100
Time (s) Time (s)
![Page 46: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/46.jpg)
Study subjects
46
Publicly AvailableName No. of
InputsHierarchical
States ParallelismNo. of States
SCPCASS
NoNo
2342
1316
2 No1 No
GCS Yes 8 10 0 Yes
No. of Transitions
2553
27
• SCPC: Supercharger Clutch Position Controller
• ASS: Auto Start Stop Control
• GCS: Guidance Control System
![Page 47: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/47.jpg)
Fault Revealing Rate (FRR)
47
FRR(SF ,TS ) =
(1 91iq
ˆdist(sgi, gi) > THR
0 81iqˆdist(sgi, gi) <= THR
• FRR based on gi, output of the fault-free model, sgi, output of the fault-seeded model, and a threshold THR:
1. For continuous dynamic systems, the system output is acceptable when the deviation is small and not necessarily zero
2. It is more likely that manual testers recognize a faulty output signal when the signal shape drastically differs from the oracle.
![Page 48: Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers](https://reader031.fdocuments.net/reader031/viewer/2022020119/58a6258d1a28ab416c8b51c5/html5/thumbnails/48.jpg)
RQ3: Test Suite Size
48
1. The fault revealing rates for output stability/continuity is very high for small test suites for Instability/Discontinuity
2. For “Other” failures, the ability of OD in revealing failures rapidly increases as the test suite size increases
Discontinuity
SCTC
ODOSOC* *+ +
--
Instability Others
0.0
0.5
1.0
3 5 10 25 50
**
* *
+
++
--
Test Suite Size
FRR
Mea
n
+-
-
3 5 10 25 50 3 5 10 25 50
* * *
-
--
+
+
+ - - -*
* *
+
++
+
+*
* *---