Generation of Random EMF Models for Benchmarks
-
Upload
markus-scheidgen -
Category
Science
-
view
24 -
download
1
Transcript of Generation of Random EMF Models for Benchmarks
Generation of Random Software Models for
BenchmarksMarkus Scheidgen
1
Agenda
▶ Benchmarks for MDE
▶ Input models for MDE benchmarks
▶ Generation of random models
■ Language
■ Examples
▶ Related Work
▶ Conclusion
2
Benchmarks (I)
▶ in smallMDE technology1 is solely evaluated by its functionality
▶ BigMDE technology is evaluated by its functionality and its performance (execution time, memory consumption, ...)
▶ Benchmarks enable sound comparison of technologies based on their performance
1) technology = algorithms ∪ methods ∪ tools ∪ frameworks
3
Benchmarks (II)
▶ A benchmark describes the measure ...
■ of a well defined property
■ acquired in a well defined processes
■ with a well defined workload (tasks and inputs)
■ in a well defined environment
4
All measurements were performed on a Notebook computer with Intel Core i5 2.4GHz CPU, 8 GB 1067 MHz DDR3 RAM, running Mac OS 10.7.3.
+ some environment- software versions, JVM
configuration
M. Scheidgen, A. Zubow, J. Fischer, T. H. Kolbe: Automated and Transparent Model Fragmentation for Persisting Large Models; ACM/IEEE 15th International Conference on Model Driven Engineering Languages & Systems (MODELS); Innsbruck; 2012; LNCS Springer
All measurements were performed on a Notebook computer with Intel Core i5 2.4GHz CPU, 8 GB 1067 MHz DDR3 RAM, running Mac OS 10.7.3.
+ some environment- software versions, JVM
configuration
M. Scheidgen, A. Zubow, J. Fischer, T. H. Kolbe: Automated and Transparent Model Fragmentation for Persisting Large Models; ACM/IEEE 15th International Conference on Model Driven Engineering Languages & Systems (MODELS); Innsbruck; 2012; LNCS Springer
+ some process- distribution, variation,
outlier- warmup: JIT, caches, GC
All experiments were repeated at least 20 times, and all present results are respective averages.
We measured the performance of instantiating and persisting objects.
+ some property description- exact task?- comparable between
technologies
All measurements were performed on a Notebook computer with Intel Core i5 2.4GHz CPU, 8 GB 1067 MHz DDR3 RAM, running Mac OS 10.7.3.
+ some environment- software versions, JVM
configuration
M. Scheidgen, A. Zubow, J. Fischer, T. H. Kolbe: Automated and Transparent Model Fragmentation for Persisting Large Models; ACM/IEEE 15th International Conference on Model Driven Engineering Languages & Systems (MODELS); Innsbruck; 2012; LNCS Springer
+ some process- distribution, variation,
outlier- warmup: JIT, caches, GC
All experiments were repeated at least 20 times, and all present results are respective averages.
We measured the performance of instantiating and persisting objects.
+ some property description- exact task?- comparable between
technologies
All measurements were performed on a Notebook computer with Intel Core i5 2.4GHz CPU, 8 GB 1067 MHz DDR3 RAM, running Mac OS 10.7.3.
+ some environment- software versions, JVM
configuration
M. Scheidgen, A. Zubow, J. Fischer, T. H. Kolbe: Automated and Transparent Model Fragmentation for Persisting Large Models; ACM/IEEE 15th International Conference on Model Driven Engineering Languages & Systems (MODELS); Innsbruck; 2012; LNCS Springer
+ some process- distribution, variation,
outlier- warmup: JIT, caches, GC
All experiments were repeated at least 20 times, and all present results are respective averages.
XMI CDO Morsa EMFFrag10
3
104
105
Ob
ject
s p
er
seco
nd
(! 1
04)
w/ cross referencesw/o cross references
+ two specific shapes- two specific shapes- real world likeness
We created test models with 105 objects, a binary containment hierarchy, and two different densities of cross references: one cross reference per object and no cross references.
Input for Benchmarks (I)
▶ A benchmark input model should
■ include no bias
■ invoke real world behavior
■ cover different scenarios
■ metrical scale
6
Input for Benchmarks – In MDE
▶ For MDE technology input is usually a software engineering artifact, which we commonly refer to as a model
▶ Usually the models from the 2009 Grabat’s graph transformation contest are used
■ MoDisco-models of JDT
■ different sizes and shapes (with and without method implementations)
■ sizes not linear
7
Input for Benchmarks – Properties: Size & Shape
▶ different properties to mimic different scenarios and invoke different behavior/performance characteristics
▶ goal: understand correlation between performance properties and model sizes and shapes
▶ ordinal vs metrical
▶ What defines a shape?
■ metrics (depending on the language, e.g. methods per class in OO programming)
■ graph/tree properties (degree, connectedness, sparse vs. dense, etc.)
▶ What defines size
■ # objects
■ # values
■ # links
8
Input for Benchmarks – Properties: Size & Shape
▶ different properties to mimic different scenarios and invoke different behavior/performance characteristics
▶ goal: understand correlation between performance properties and model sizes and shapes
▶ ordinal vs metrical
▶ What defines a shape?
■ metrics (depending on the language, e.g. methods per class in OO programming)
■ graph/tree properties (degree, connectedness, sparse vs. dense, etc.)
▶ What defines size
■ # objects
■ # values
■ # links
9
Input for Benchmarks – Approaches
▶ handcraft input models – no scalability
▶ take existing models – only given shapes
▶ generate models – do not mimic the real world
▶ bias?
■ bias in creation, selection, algorithm
■ social problem, can’t use technology to solve social problems
10
Input for Benchmarks – Random Models
▶ random ≠ arbitrary nor uniform
▶ surprise element
▶ probability distributions as abstractions for typical usage of language constructs
■ e.g. a class has typically a negative binomial distributed (with certain parameters) number of methods [1]
▶ distribution parameters to define shapes
▶ random models can be sensible representatives of a large class of models
11
[1] Tetsuo Tamai, Takako Nakatani: Analysis of Software Evolution Processes Using Statistical Distribution Models, IWPSE '02
Generation of Random Models – A Generator DSL (I)
12
generator RandomEcore for ecore in "...ecore/model/Ecore.ecore" { ePackage: EPackage -> name := RandomID(Normal(8,3)) eClassifiers += eClass#NegBinomial(5,0.5) ; eClass: EClass -> name := RandomID(Normal(10,4)) abstract := UniformBool(0.2) eStructuralFeatures += eReference(UniformBool(0.3))#NegBinomial(4,0.7) eStructuralFeatures += eAttribute#NegBinomial(6,0.5) ; eReference(boolean composite):EReference -> name := RandomID(Normal(10,4)) upperBound := if (UniformBool(0.5)) -1 else 1 ordered := UniformBool(0.2) containment := composite eType:EClass := Uniform(model.EClassifiers.filter[it instanceof EClass]) ;
...}
http://github.com/markus1978/RandomEMF
Input for Benchmarks – A Generator DSL (II)
13
▶ Maps Meta-Model to Grammar-like description
▶ Rule based
▶ Each rule creates an object of a certain meta-class
▶ Each rule calls other rules to create features
▶ Rules can have parameters
▶ Expressions with random values
■ different distributions for random number generation
■ random number of rule application
■ random values (e.g. identifier, choices)
▶ xText + xBase DSL
Generation of Random Models – Generated Example
14
1. package dabobobues;
3. class Dues {4. 5. DuBoBuTus begubicus;6. ELius brauguslus;7. 8. void Dues(Alius donus, FanulAudaCio aubetin) {9. }10. 11. void baGusFritus() {12. eudaguslius = "";13. bigusdaGubolius();14. if ("") {15. annulAugusaugusfrigustin("");16. albucio = Dues()<=++12;17. bi();18. eBoTor();19. } else {20. brauguslus = 9;21. baGusFritus();22. duLus = ""=="";23. }24. }25. 26. void aufribonulAubufrinus(Dues e) {27. dobubogutor();28. aubiguTus = 9;29. }30. }
Classes/InterfacesMethods
StatementsExpressions
others
randomly generated code
actual Java code
syntheticly generated code
15
15
15
generatedgenerated
actual project
actual project
generatedgenerated
Generation of Random Models – Problems
▶ randomness is a tool to reduce bias, but clients have to decide to use it correctly
▶ hard to generate static semantically correct models
16
Related Work
▶ Test-Model generation with SAT-Solvers
■ Meta-Model/Constraint divided into small partitions that cover test-cases
■ translation into logical equations
■ SAT-Solver
■ translation of results into model-fragments
■ composition of test-models from model fragments
➡ small, valid models with statistically proved test-coverage
17
Sagar Sen, Benoit Baudry, Jean-Marie Mottu: Automatic Model Generation Strategies for Model Transformation Testing, Theory and Practice of Model Transformations, Springer, 2009
Erwan Brottier, Franck Fleurey, Jim Steel, Benoit Baudry, Yves Le Traon: Metamodel-based Test Generation for Model Transformations: an Algorithm and a Tool, ISSRE’06, IEEE, 2006
Related Work
▶ Translation into a constructive formalism
■ Meta-Modeling is not constructive (full set of instances can not be generated from a meta-model)
■ translation into context-free or graph-grammars
■ random application of rules to generate random models
➡ large models, shape can be influenced via probability distributions on rule selection
18
K Ehrig, JM Küster, G Taentzer: Generating instance models from meta models, Formal Methods for Open Object-Based Distributed Systems, Springer, 2006
Related Work
▶ Fitting meta-model instances onto randomly generated tree/graph structures
■ existing methods for random tree or graph generation
■ interpretation of randomly generated trees/graphs as meta-model instances
➡ large models, but uniform models, not static semantic aware
19
A Mougenot, A Darrasse, X Blanc, M Soria: Uniform random generation of huge metamodel instances, ECMDA, Springer, 2009
Related Work
▶ benchmark definitions for graph transformations
▶ different distribution for graph edges to create different shapes
■ binomial
■ hypergeometric
■ uniform
■ preferential attachment
➡ large models, not static semantic aware
20
Izso, B., Szatmari, Z., Bergmann, G., Horvath, A., & Rath, I.: Towards precise metrics for predicting graph query performance. 2013 28th IEEE/ACM International Conference on Automated Software Engineering, ASE 2013
Conclusions
▶ benchmarking in MDE can be improved
▶ there are other options for input models than the Grabats’ 09 contest models
▶ different shapes (preferably on a metrical scale) should be used to find distinctive merits and flaws in compared technologies
▶ generators for random models
■ parameters to create differently shaped models
■ randomness and suitable distributions for real world like input
■ linear scaled sizes21