Automated Model-Based Testing of Hybrid Systems Michiel van Osch PROSE January 25, 2007 13.
Automated Model-Based Testing of Hybrid Systems
description
Transcript of Automated Model-Based Testing of Hybrid Systems
![Page 1: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/1.jpg)
Automated Model-Based Testing of Hybrid Systems
Michiel van Osch
PROSEJanuary 25, 2007
![Page 2: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/2.jpg)
2
Motivation• Hybrid Systems
• Testing might be expensive, dangerous, or resources might be limited
• Discrete and real-time model-based testing does not test the continuous aspects of the system
![Page 3: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/3.jpg)
3
Content
• Part I: Theory– Model-based Testing– Input-Output Conformance & Discrete Tests– Hybrid Systems– Hybrid Input-output Conformance– Hybrid Tests– Results
• Part II:Tool– Test Architecture– Specification– Tester– The Connection with the Implementation Under Test– Adapter– Limitations and Future Work– Case Study: Vacuum Control
![Page 4: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/4.jpg)
4
Model-Based Testing
model
SUT
IUTconforms to
model
pass fail
test
tool
testgeneration
tool
test execution
toolIUT passes tests
IUT conf model
soundexhaustiv
e
![Page 5: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/5.jpg)
5
Input-output Conformance for Discrete Systems
Impl. ioco Spec. iff for all traces α:
out(Impl. after α) out(Spec. after α)
If there is an output action from state s thenout(s) = {o in O| s →}
elseout(s) = {δ}
Furthermore,out(S) = UsS out(s)
![Page 6: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/6.jpg)
6
Test-case Generation and Execution
1. Terminate with verdict pass
2. Select an input from the specification and apply it to the implementation
3. Observe an output or a timeout from the implementation and check if it is allowed according to the specification
![Page 7: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/7.jpg)
7
Example
s1s0
s2
s3
?Activate
?Button2
?Button1
!XLCoffee
!Coffee
t0
t1
?Activate
t3
?Button2
t2
! δ
FailFail
!Coffee!XLCoffee
Pass
!XLCoffee
FailFail
!Coffee! δ
![Page 8: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/8.jpg)
8
Hybrid Systems
• In Practice:– Discrete behavior plus continuous behavior– Continuous behavior can be input observed through sensors
or output generated by actuators
• In Theory:– Discrete actions plus flow of continuous variables
(trajectories)– Variables can be input variables and output variables– Hybrid Transition Systems
![Page 9: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/9.jpg)
9
Hybrid Systems (Output Only)
Time
Coffee
0
?Button1 ?Button2
s1s0
s2
s3
?Activate
?Button2
?Button1
Coffee’ = 4 cl/sec.Δt = 8 sec.
Coffee’ = 3 cl/sec.Δt = 5 sec.
Coffee’ = 0Δt = 1 sec.
Coffee’ = 0Δt = 1 sec..
![Page 10: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/10.jpg)
10
Hybrid Systems (Including Input)
Water
s1s0
s2
s3
?Activate
?Button2
?Button1
Water’ = 4 cl/sec.Coffee’ = Water’Δt = 8 sec.
Water’ = 3 cl/sec.Coffee’ = Water’Δt = 5 sec.
Water’ = 0Coffee’ = 0Δt = 1 sec.
Water’ = 0Coffee’ = 0Δt = 1 sec..
![Page 11: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/11.jpg)
11
Hybrid Conformance
• For every reachable state, the set of output actions possible by the implementation is a subset of the set of output actions possible by the specification
• For every reachable state, the set of trajectories possible by the implementation is a subset of the set of trajectories possible by the specification
• In contrast to ioco, no quiescence action because there is always continuous output.
![Page 12: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/12.jpg)
12
Continuous Output Only
Impl. is input-output conform a Spec. iff for all traces α:
out(Impl. after α) out(Spec. after α) and
traj(Impl. after α) traj(Spec. after α)
s1s0
s2
s3
?Activate
?Button2
?Button1
Coffee’ = 4 cl/sec.Δt = 8 sec.
Coffee’ = 3 cl/sec.Δt = 5 sec.
Coffee’ = 0Δt = 1 sec.
Coffee’ = 0Δt = 1 sec..
![Page 13: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/13.jpg)
13
With Continuous Input
• The implementation is input enabled (for both discrete behavior and continuous behavior).
• We do not require the specification to be input complete.
Impl. is input-output conform a Spec. iff for all traces α:
out(Impl. after α) out(Spec. after α) and
traj(Impl. after α) traj(Spec. after α)
Does not work!!
Solution: Look at the trajectories of the Implementation with respect to the trajectories of input variables of the Specification
![Page 14: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/14.jpg)
14
Hybrid Conformance (Continuous Input plus Output)
infilter(traj(Impl. after α), traj(Spec. after α)) traj(Spec. after α)
s1
s3
?Button2
Water’ = 0Coffee’ = 0Δt = 1 sec..
s4Water’ = 0Coffee’ = 0Δt = 3 sec..
s5!Out of Cups
Water’ = 0Coffee’ = 0Δt = 1 sec..
u1
u3
?Button2
Water’ = 0Coffee’ = 0Δt = 1 sec..
u4Water’ = 0Coffee’ = 0Δt = 3 sec..
u5!Out of Cups
Water’ = 0Coffee’ = 0Δt = 1 sec..
Specification Implementation
Water’ = 0Coffee’ = 0Δt = 1 sec..
Still does not work because …
![Page 15: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/15.jpg)
15
Hybrid Conformance (continuous input plus output)
If there is a trajectory from state s thenout(s) = {o in O| s →} {ξ}
elseout(s) = {o in O| s →}
Impl. hioco Spec. iff for all traces α:
out(Impl. after α) out(Spec. after α) and
infilter(traj(Impl. after α), traj(Spec. after α)) traj(Spec. after α)
![Page 16: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/16.jpg)
16
Hybrid Tests
A Special kind of Hybrid Transition Systems:• Tree like structure• Two terminal states: pass and fail• Deterministic for actions• Strongly time deterministic for trajectories
![Page 17: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/17.jpg)
17
Hybrid Tests
1. Terminate with verdict pass
2. Select an input from the specification and apply it to the implementation
s1s0?Activate
Water’ = 0Coffee’ = 0
Specification
t0
t1
?Activate
Test
![Page 18: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/18.jpg)
18
Hybrid Test-case Generation
3. If an output action has to happen immediately according to the specification then observe an output action and check if it is allowed according to the specification or let time pass by selecting and applying and observing a trajectory
Fail Fail
s4 s5!”Out of Cups”
Coffee’ = 0Water’ = 0Δt = 1
Specification
t4
Pass
!”Out of Cups”
Test-Case
Coffee’ = 0Water’ = 0Δt = 1
![Page 19: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/19.jpg)
19
Hybrid Test-case Generation
4. Select an input trajectory from the specification, apply it to the implementation and observe the output trajectory simultaneous, possibly interrupted by an output action.
Specification
s1Water’ = 0Coffee’ = 0Δt = 1
Test
t1
t2
Water’ = 0Coffee’ = 0Δt = 1
Fail Fail
Fail Fail
!”out of cups”Water’ = 0
Coffee’ = 3Δt = 1
![Page 20: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/20.jpg)
20
Results
• A hybrid conformance theory• Proven Sound and exhaustive• A Natural extension of discrete and timed
conformance theories
![Page 21: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/21.jpg)
21
Content
• Part I: Theory– Model-based Testing– Input-Output Conformance & Discrete Tests– Hybrid Systems– Hybrid Input-output Conformance– Hybrid Tests– Results
• Part II: Tool– Test Architecture– Specification– Tester– The Connection with the Implementation Under Test– Adapter– Limitations and Future Work– Case Study: Vacuum Control
![Page 22: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/22.jpg)
22
Libraries
Tester Architecture
• Specification: The Model from which Tests are Generated• Tester: Implements The Test Algorithm and Gives the Verdict• Adapter: Translated Input/Output from Model to a format suitable
for the Implementation Under Test and vice versa• Medium: The Interface between Tester and Implementation• IUT: The Implementation Under Test
Spec Tester Adapter
Medium IUT
![Page 23: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/23.jpg)
23
Libraries
Specification
Needs to:• Model Discrete behavior and Continuous Behavior• Make Distinction between Input Actions, Output
Actions, and Internal Actions• Make Distinction between Input Variables, Output
Variables and Internal Variables• Model in an Intuitive way
Medium
AdapterSpec Tester IUT
![Page 24: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/24.jpg)
24
Libraries
Specification
Medium
AdapterSpec Tester
proc Control(cont V: real, chan h,out: real)=|[ *(V <= 2 -> h!!1.0; out!!1.0; V >= 10 -> h!!0.0; out!!0.0)]|
proc Env(cont V: real, chan h: real)=|[ var n: real = 0.0:: V’=3.0*n - 1.0| *(h?n)]|
model Spec()=|[ cont V: real = 10.0, chan h,out: real:: Control(V,h,out)|| Env(V, h)]|
IUT
![Page 25: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/25.jpg)
25
Libraries
Tester
Implements:• On the Fly Test Generation
– Select Input from Specification– Apply Input– Observe Output– Compare the Observed Output with the Output allowed by the
Specification– Give a Verdict or Continue Test
Medium
AdapterSpec Tester IUT
![Page 26: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/26.jpg)
26
On the Fly Testing
proc ControlS(cont V: real, chan h: real)=|[ var n: real = 0.0:: V’=3.0*n - 1.0| *(V <= 2 -> n:=1.0; h!!1.0; V >= 10 -> n:=0.0; h!!0.0)]|
model Spec()=|[ cont V: real = 10.0, chan h: real:: ControlS(V,h)]|
V=10
V=2
V’=3.0*0.0-1.0Δt = 8 sec.
Select Input(χ)(Manually/ Automatic)
V=2
h!!1.0
passfail
h!!0.0
CompareValues(χ , Maple)
Pass
Give Verdict(with trace)
Continue
Apply(Via adapter)
Pressure’= -1.0 mbar/secΔt = 8 sec.
IUT
Pump OFF
Observe(Via adapter)
IUT
![Page 27: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/27.jpg)
27
Additional Libraries
• χ –stepper for computing sets of allowed transitions and current state of the specification
• E.g. Maple for comparing observed continuous output (samples) with specified trajectories and comparing observed discrete output values with specified send actions
Libraries Medium
AdapterSpec Tester IUT
![Page 28: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/28.jpg)
28
Libraries
The Connection
Medium
Jabber χ Model
TCP/IP Labview Controller
Wires Electronics
Buttons/Sensors
Robot Arm
AdapterSpec Tester IUT
![Page 29: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/29.jpg)
29
Libraries
Tester
The Adapter
• Implements– Mapping of Variables/Actions of Specification to a
Implementation and vice versa (e.g. channels to function calls , or variables to wires)
– Translating Input/Output of Specification to Implementation and vice versa (e.g. functions to samples, or signals)
Medium
AdapterSpec IUT
![Page 30: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/30.jpg)
30
Limitations and Future Work
• This is just a prototype, there are shortcomings!– Real Time Testing is Not Possible Yet– The complexity of Continuous behavior is limited by the
Hybrid χ –stepper implementation. E.g. currently only standard differential equations.
– Models are not ‘ideal’ for testing. E.g. in case of identifying input and output
– For performance reasons we only deal with deterministic specifications.
– We assume that the communication medium is reliable
• Adaptation of theory for Sampling and Inaccuracy• Case Studies
![Page 31: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/31.jpg)
31
Real Time
• Generating and applying input (e.g. samples)
• Observing output and Time at which output Occurred in the Implementation
![Page 32: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/32.jpg)
32
Limitations and Future Work
• This is just a prototype, there are shortcomings!– Real Time Testing is Not Possible Yet– The complexity of Continuous behavior is limited by the
Hybrid χ –stepper implementation. – Models are not ‘ideal’ for testing. – For performance reasons we only deal with deterministic
specifications.– We assume that the communication medium is reliable
• Adaptation of theory for Sampling and Inaccuracy• Case Studies
![Page 33: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/33.jpg)
33
The Vacuum Case
• Lithography Process takes place in vacuum
• Waferstepper has Five Chambers
• Chambers are kept in Vacuum by a system of Pumps and Valves
• Pumps and Valves are Controlled by Software (discrete)
• Software observes Pressure in Chambers through Sensors (continuous)
![Page 34: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/34.jpg)
34
Activities
• Modeling Hardware in Hybrid χ and Stand Alone Simulation
• Modeling (translating) Hardware in discrete (timed) χ and Integration with Software Controller
• Modeling (translating) in Uppaal for Model Checking• Testing Models and Software Controller with the
Hybrid Tester
![Page 35: Automated Model-Based Testing of Hybrid Systems](https://reader035.fdocuments.net/reader035/viewer/2022062803/568146fa550346895db433ce/html5/thumbnails/35.jpg)
35
Questions?