Research Questions for Validation and Verification in the Context of Model-Based Engineering
-
Upload
michalis-famelis -
Category
Technology
-
view
257 -
download
2
description
Transcript of Research Questions for Validation and Verification in the Context of Model-Based Engineering
![Page 1: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/1.jpg)
Research Questions for Validation and Verification in the Context of Model-Based
Engineering
Catherine DuboisMichalis FamelisMartin GogollaLeonel NobregaIleana OberMartina SeidlMarkus Voelter
ENSIIE, Evry, FranceUniversity of Toronto, University of Bremen, University of Madeira, Funchal, University of Toulouse, Johannes Kepler University Linz, AustriaVoelter Ingenieurburo, Heidenheim, Germany
October 1st , 2013MoDeVVa 2013, Miami, USA
![Page 2: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/2.jpg)
2
Introduction
Abstraction techniques are one of the promising paths for the future advances in the field of verification.
– Clarke, Emerson, Sifakis: Turing Lecture 2008
V&V crucial for MBE– Uncover hidden properties and errors– Verify transformations– Ensure quality, etc.
![Page 3: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/3.jpg)
3
About
• This paper is the result of the working group on V & V at Dagstuhl-Seminar 13182 held in May 2013.
• Dagstuhl: scientific retreatin western Germany
• Seminar topic: Meta-Modeling Model-Based Engineering Tools
• Three break-outs: Informal Modeling, Compositionality, Modeling and V&V
![Page 4: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/4.jpg)
4
Our goal:Identify main areas in the synergy between MBE and V&V where we need to focus research. What is the status, what are the research questions?
• Two-day collaborative brainstorming workshop.
• Culminated in a plenary presentation
…and this report
![Page 5: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/5.jpg)
5
Disclaimer
• No claim of completeness or exhaustiveness
• Represents the informed opinions of the authors
• May have missed existing answers to some questions
• Is this the right level of granularity / level of detail?
![Page 6: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/6.jpg)
6
Thematic Categories
1. Gap between Models and V&V Formalisms2. Need to Refine Existing Methodologies3. Design-time vs. Runtime4. Properties5. Model Transformations6. Informal vs. Formal vs. Incomplete Modeling7. Comparison and Benchmarking8. Domain-Specific Languages
![Page 7: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/7.jpg)
7
Models and V&V Formalisms (context)
Model Property
Model' Property'
Designer Level (can be Domain-Specific)
Verification formalism level
Verification engine
Verification
feed-back
Transformationsand traceability
mechanisms
But behavioral semantics often leads to non-bijective correspondences between design time and runtime artifacts.
V&V tool configuration
![Page 8: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/8.jpg)
Models and V&V Formalisms (questions)
• How to express properties at the level of models in a way understandable to clients?
• How to formulate models and properties in a single language transparent to clients?
• How to report the V&V results and diagnostics in an appropriate form to clients?
• How to bridge the gap between formally expressed and verified properties on one side and client attention on the other side?
• Can modeling language extensions help in making explicit the “needs” of V&V machines?
![Page 9: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/9.jpg)
9
Refining Existing Methodologies (context)
• Integrating V&V in development can support early V&V similar to how debugging is offered by IDEs.
• Generic methodologies identify points where V&V can be used.
• A Model-based development methodology would need to allow variations based on application domain, nature of project, etc.
• Goal: better identify which V&V activities are meaningful at the various phases of design; take full advantage of V&V engines.
![Page 10: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/10.jpg)
10
Refining Existing Methodologies (questions)
• How do we integrate V&V in the overall development and modeling process? – On the technical level of tool exchange? – On the methodological level of using the right technique at
the right time for the right task?
• When are techniques like animation, execution, symbolic evaluation, testing, simulation, proving or test case generation used most efficiently during development?– For which model and model transformation properties can
they be employed?
![Page 11: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/11.jpg)
11
Design-time vs. Runtime (context)
• Models are specified at design time
• During execution these models are instantiated
• The dynamic nature of structure during execution makes it difficult to understand and represent runtime information
• Existing modeling environments offer limited support for precisely specifying instantiation and snapshots
![Page 12: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/12.jpg)
12
Design-time vs. Runtime (questions)
During the V&V phase, how do we obtain an initial model instantiation?
How do we obtain large and meaningful instantiations?
How do we connect design time and runtime artifacts?
How do we deal with the issue of scalability in the context of V&V ?
How do we handle time and space concerns w.r.t. design time and runtime artifacts?
How do we automatically or semi-automatically manage the V&V machine configuration?
![Page 13: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/13.jpg)
13
Properties (context)• Model and model transformation properties relevant in V&V:
– consistency, reachability, dependence, minimality, conformance, safety, liveness, deadlock freeness, termination, confluence, correctness
• Confusion caused by: – Different kinds according to the nature of the model (static/ dynamic),
its level of abstraction, etc.– Many tools and techniques (potentially complementary)
Main challenge: What kind of property to verify on which model at what stage with what kind of technique?
![Page 14: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/14.jpg)
14
Properties (questions)• What are the benefits and trade-offs between expressing properties
on more abstract modeling levels in contrast to expressing them on more concrete levels?
• How do we find the right techniques for uncovering static and dynamic model properties?
• Which techniques are appropriate for uncovering static modeling language inherent properties, which for static model-specific properties?
• Which techniques are appropriate for uncovering dynamic generic properties, which for dynamic model-specific properties?
![Page 15: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/15.jpg)
15
Model Transformations (context)
• Core component of MBE• Many applications:
– Maintaining inter-model consistency– Semantics definition of (domain-specific) modelling languages– …
• Challenge: Verification of model transformations– Proving correctness, termination, confluence– What are the differences to “normal” code?– Is the higher abstraction level beneficial to V&V?
![Page 16: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/16.jpg)
16
Model Transformations (questions)
• What verification techniques are meaningful for verifying model transformations?
• How do we analyse properties like confluence and termination?
• How do we analyse correctness of model transformations w.r.t. a transformation contract?
• How do we infer a transformation contract from a model transformation?
![Page 17: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/17.jpg)
17
Informal vs. Formal vs. Incomplete Modeling (context)
• During V&V: switch on or off particular model elements (in class diagrams, e.g., multiplicities); configure constraints by negating, deactivating or activating them (in class diagrams, e.g., class invariants)
• Different types of granularity (a) all model elements may be relaxed (b) only a manual model element selection can be considered for relaxation (c) a semi-automatic element selection for relaxation may be offered
• Ultimate vision: sliders on the user interface to gradually go from a strict, formal model through various intermediate levels to a totally relaxed and informal model; fewer formal model parts activated means more informality in the model
• Minimal formal frame for test case construction must be preserved: e.g., for class diagrams of central classes and associations and for state charts of central states and transitions
• For a completely informal model no formal scenario (no test cases) can be formulated
![Page 18: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/18.jpg)
18
Informal vs. Formal vs. Incomplete Modeling (questions)
• How do we leverage informal assumptions found in sketches for exploratory V&V?
• Are informal sketches close enough to V&V at all?
• What are appropriate relaxation mechanisms for different degrees of formality?
• How do we handle incomplete or partial models w.r.t. V&V?
• How do we deactivate and activate model units?
• How do we handle the exploration of model properties and alternatives?
![Page 19: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/19.jpg)
19
Comparison and Benchmarking (context)
• Benchmarking can boost research:– Fair comparison of tools– Impartial benchmark selection which covers the spectrum of
interesting test cases– Clear documentation of outcomes; reproducibility– Publicity makes it easy to identify progress, problems
• However, in MBE:– No common standards– No community platform for benchmarks– Not clear what metrics are relevant for measuring “improvement”
![Page 20: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/20.jpg)
20
Comparison and Benchmarking (questions)
• How to compare existing V&V tools w.r.t. functionality, coverage, scalability, expressiveness, executing system (i.e., for models at runtime)?
• Which criteria are appropriate for comparison?
• Can we globally compare fairly at all?– Broad and diverse spectrum of V&V machines:
B, Coq, HOL/Isabelle, SAT, SMT, CSP solvers, Relational logic, enumerative techniques
![Page 21: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/21.jpg)
21
Domain-Specific Languages (Context)
• Most verification tools have hard to use input languages, alien to normal developers.– Hence, verification tools are often not used.
• MBE approaches become more and more mainstream.
• Models can simplify analysis and verification, because of the higher degree of domain semantics they express.
• Potential to exploit the two approaches synergistically– From the high-level models, we can automate the generation of
the input to the verification tools.
![Page 22: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/22.jpg)
22
Domain-Specific Languages (Questions)
• How to define DSLs close to domain concepts but still allow generation of meaningful input for V&V tools?
• V&V tools need the specification of properties. – How to express them at the domain level in a user-friendly way? – Can the property specifications be integrated with the same DSL and/or
model used for describing the to-be-verified system without creating self-fulfilling prophecies?
• How to bring V&V feedback back to the domain level and express it in terms of the DSL-level input?
• Can incremental languages extensions help with making programs expressed in general-purpose languages more checkable? For example, the semantics of a specific extension construct may enable the generation of very rich inputs to the verification tool.
![Page 23: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/23.jpg)
23
Thematic Categories
1. Gap between Models and V&V Formalisms
2. Need to Refine Existing Methodologies3. Design-time vs. Runtime4. Properties5. Model Transformations6. Informal vs. Formal vs. Incomplete
Modeling7. Comparison and Benchmarking8. Domain-Specific Languages
![Page 24: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/24.jpg)
24
Conclusion
• Report-back from working group on MBE and V&V at Dagstuhl in May 2013.
• Main areas in the synergy of MBE and V&V.
• For each: described status and identified research questions.
• Our hope:
Spark discussions and debate, help focus research in MBE and V&V.
![Page 25: Research Questions for Validation and Verification in the Context of Model-Based Engineering](https://reader035.fdocuments.net/reader035/viewer/2022070315/554f6d09b4c9058a148b501c/html5/thumbnails/25.jpg)
25
Questions?