Testing the Untestable: Model Testing of Complex Software-Intensive Systems
-
Upload
lionel-briand -
Category
Software
-
view
215 -
download
0
Transcript of Testing the Untestable: Model Testing of Complex Software-Intensive Systems
![Page 1: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/1.jpg)
Testing the Untestable: Model Testing
of Complex Software-Intensive Systems"
Lionel Briand joint work with Shiva Nejati, Mehrdad Sabetzadeh, Domenico Bianculli
![Page 2: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/2.jpg)
Untestable Systems
2
“systems for which testing is highly expensive "or impossible to fully automate”
![Page 3: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/3.jpg)
Untestable Systems"Test Generation
3
large, complex input spaces
![Page 4: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/4.jpg)
Untestable Systems"Test Execution
4
hardware-in-the-loop
![Page 5: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/5.jpg)
Untestable Systems"Identification of failures
5
dynamic behavior
• Inputs are signals over time
• Properties to be checked consider change over time, for independent or related output signals
• Common when dealing with physical entities
Initi
al D
esire
d(ID
)
Desired ValueI (input)Actual Value (output)
Fina
l Des
ired
(FD
)
timeT/2 T
Smoothness
Responsiveness
Stability
![Page 6: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/6.jpg)
Untestable Systems
6
limited testing
only a small number of test scenarios "can be executed and analyzed "
within time and budget constraints
![Page 7: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/7.jpg)
Cyber-physical Systems as Untestable Systems
7
![Page 8: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/8.jpg)
Testing Cyber-physical Systems: Challenges
8
complex systems
hybrid behavior
complex"environment
combinatorial"state
explosion complex"
requirements uncertainty
![Page 9: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/9.jpg)
Example: Night Vision System
9
camera-based assistance system"providing improved vision at night
Large, complex input space: • traffic situations • shape, speed, "
and distance • lightning, weather
![Page 10: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/10.jpg)
Example: Controllers
10
• how to select test cases from the vast space of possible input signals? • how to automatically distinguish faulty from non-faulty output signals?
![Page 11: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/11.jpg)
Software V&V: Common Model-based Approaches
11
Model-based"Testing
Model"Checking
![Page 12: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/12.jpg)
Model-based Testing
• Relies on models to generate test scenarios and oracles for the implemented system
• Has challenges with:
• hardware-in-the-loop
• uncertainty
• hybrid, discrete-dynamic behavior 12
![Page 13: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/13.jpg)
Model Checking
• Verification of temporal properties over concurrent, finite state models
• Has challenges with:
• modeling complex properties of physical devices with continuous dynamics
• modeling complex, concurrent interactions with the environment
• Not easy to parallelize 13
![Page 14: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/14.jpg)
Software V&V: Common Model-based Approaches
14
Model-based"Testing
Model"Checking
Scalability and practicality challenges in the context of untestable systems!!!
![Page 15: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/15.jpg)
Model Testing
![Page 16: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/16.jpg)
Model Testing
16
Raise the level of abstraction"at which (most of the) testing is performed
![Page 17: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/17.jpg)
Model Testing
17
Shift the bulk of testing"from implemented systems "
to models of such systems and their environments
![Page 18: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/18.jpg)
Model Testing: Strategy
• Make the best out of available time and resources for testing the implemented system
• Focus on high-risk test scenarios within budget and time constraints
18
![Page 19: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/19.jpg)
• Capture relevant aspects of system behavior, structure, environment, and properties, as well as their uncertainty
• Are used as a basis for execution and risk analysis
• Use the results to focus the testing of the implemented system
The Role of Models
19
![Page 20: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/20.jpg)
Model Testing vs. "Model-based Testing
20
The focus of model testing is on"testing the model,
not the implementation
![Page 21: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/21.jpg)
Model Testing vs."Model Checking
• Not exhaustive, but risk-driven
• Oracles can be dynamic properties over time intervals and handle uncertainty
• Supports modeling the physical world, heterogeneous formalisms and integration languages
21
![Page 22: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/22.jpg)
Model Testing: Objectives and Challenges (1/5)
22
Executable and testable models of the system and
its environment
Objective Definition and execution of testable models, including
dynamic and uncertain behavior
Challenge
![Page 23: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/23.jpg)
Model Testing: Objectives and Challenges (2/5)
23
Finding, executing, "and analyzing "
high-risk test scenarios
Objective Automated, effective test selection in large input
spaces, partly defined by input signals
Challenge
![Page 24: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/24.jpg)
Model Testing: Objectives and Challenges (3/5)
24
Modeling"uncertainty and risk
Objective
Automated detection of failures under uncertainty;
risk assessment
Challenge
![Page 25: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/25.jpg)
Model Testing: Objectives and Challenges (4/5)
25
Change impact analysis "and regression testing
Objective
Regression tests prioritization or selection
when models change
Challenge
![Page 26: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/26.jpg)
Model Testing: Objectives and Challenges (5/5)
26
Guiding testing on the implemented system
(with HiL)
Objective
Selecting test cases to run so as to reduce overall risk
Challenge
![Page 27: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/27.jpg)
Other Questions
27
• Cost of modeling
• Fidelity of modeling and its impact on cost-effectiveness
• Impact of uncertainty on finding high-risk scenarios and fault detection
![Page 28: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/28.jpg)
Model Testing
28
Shift the bulk of testing"from implemented systems "
to models of such systems and their environments
Accounts for uncertainty, dynamic behavior, a diversity of modeling languages from different
domains, risk modeling Better focus testing of implemented systems"
on high-risk test scenarios within limited resources
![Page 29: Testing the Untestable: Model Testing of Complex Software-Intensive Systems](https://reader033.fdocuments.net/reader033/viewer/2022051523/58ad18ad1a28ab50728b4d63/html5/thumbnails/29.jpg)
Testing the Untestable: Model Testing
of Complex Software-Intensive Systems"
Lionel Briand joint work with Shiva Nejati, Mehrdad Sabetzadeh, Domenico Bianculli
We’re hiring!!!