Post on 03-Aug-2015
www.unamur.be
Covering SPL Behaviour with Sampled Configurations: An Initial Assessment
Xavier Devroey <xavier.devroey@unamur.be> Gilles Perrouin <gilles.perrouin@unamur.be>Axel Legay <axel.legay@inria.fr>Pierre-Yves Schobbens <pierre-yves.schobbens@unamur.be>Patrick Heymans <patrick.heymans@unamur.be>
9th International Workshop on Variability Modelling of Software-intensive Systems, VaMoS ‘15
Hildesheim, Germany
2
Introduction
• Software Product Line Model Based Testing• Structural approaches– Sample configurations to test in a Feature Diagram– Similarity – Pairwise
• Behavioural approaches– Select test cases in a behavioural model of the SPL
VaMoS '15 xavier.devroey@unamur.be
Devroey et al. (2014). Towards Statistical Prioritization for Software Product Lines Testing. In VaMoS ’14.
3
Introduction
• Software Product Line Model Based Testing• Structural approaches– Sample configurations to test in a Feature Diagram– Similarity – Pairwise
• Behavioural approaches– Select test cases in a behavioural model of the SPL
Practically, which behavioural coverage similarity and pairwise sampling do achieve ?
VaMoS '15 xavier.devroey@unamur.be
SPECIFYING A PRODUCT LINE
Featured Transition Systems (FTSs) [Classen et al. 2011]
xavier.devroey@unamur.be 5VaMoS '15
Images credit: http://hedbonstudios.deviantart.com
xavier.devroey@unamur.be 6VaMoS '15
Images credit: http://hedbonstudios.deviantart.com
xavier.devroey@unamur.be 7
Featured Transition System
VaMoS '15
STRUCTURAL VS BEHAVIOURAL COVERAGE CRITERIA
Image credits: http://domonated.deviantart.com
xavier.devroey@unamur.be 9
Structural Coverage
• Which configurations do I select in the FD ?• Pairwise criterion– All pairs of features appears in at least one configuration to test– T-wise – 3-wise interactions are common [Steffens et al. 2012]
VaMoS '15
xavier.devroey@unamur.be 10
Structural Coverage
• Which configurations do I select in the FD ?• Pairwise criteria– All pairs of features appears in at least one configuration to test– T-wise – 3-wise interactions are common [Steffens et al. 2012]
• (Di)Similarity criteria– Dissimilar configurations to test– With few selected/deselected features in common– Uses a distance function (easy to compute)– Empirically validated [Henard et al. 2013]
VaMoS '15
Steffens et al. (2012). Industrial Evaluation of Pairwise SPL Testing with MoSo-PoLiTe. In VaMoS ‘12
xavier.devroey@unamur.be 11
Structural Coverage
VaMoS '15
xavier.devroey@unamur.be 12
Structural Coverage
VaMoS '15
xavier.devroey@unamur.be 13
Behavioural Coverage
• Behavioural test case – Sequence of actions in a FTS– Executable by a valid product of the SPL
VaMoS '15
Devroey et al. (2014). Coverage Criteria for Behavioural Testing of Software Product Lines. In ISoLA’1
xavier.devroey@unamur.be 14
Behavioural Coverage
VaMoS '15
xavier.devroey@unamur.be 15
Behavioural Coverage
VaMoS '15
xavier.devroey@unamur.be 16
Behavioural Coverage
VaMoS '15
xavier.devroey@unamur.be 17
Behavioural Coverage
• Behavioural test case – Sequence of actions in a FTS– Executable by a valid product of the SPL
• Coverage criteria– Executable set of test cases, FTS → [0, 1]– State coverage– Transition coverage – Transition-pairs coverage
VaMoS '15
Devroey et al. (2014). Coverage Criteria for Behavioural Testing of Software Product Lines. In ISoLA’14
xavier.devroey@unamur.be 18
Behavioural Coverage
Exec. Test-Suite TS is Minimal for cov iff
• There is no exec. TS’ with a better coverage cov
• There is no exec. TS’ with fewer Test-Cases
Exec. Test-Suite TS is P-Minimal for cov iff
• There is no exec. TS’ with a better coverage cov
• There is no exec. TS’ which request fewer products to execute TS’
SPLat @ SPLC'14
Devroey et al. (2014). Coverage Criteria for Behavioural Testing of Software Product Lines. In ISoLA’14
xavier.devroey@unamur.be 19
FTS Test-Case Selection
Number of Exec T.
C.
Number of
Products
Test Case Selection
Coverage
SPLat @ SPLC'14
xavier.devroey@unamur.be 20
Behavioural Coverage
VaMoS '15
xavier.devroey@unamur.be 21
Behavioural Coverage
VaMoS '15
EXPERIMENT
… using PLEDGE and SPLCAT
xavier.devroey@unamur.be 23
Setup
• SPLCAT– T-wise criterion, with T in [1,3]
• PLEDGE– (Di)Similarity criterion– Uses a genetic algorithm
• Number of configurations (population) given by SPLCAT• Time = 60 sec. (default time)
VaMoS '15
xavier.devroey@unamur.be 24
Coverage computation process
Compute coverage and cumulated coverage
Projects the FTS Model
Generate a set of configurations from the FD
VaMoS '15
xavier.devroey@unamur.be 25
Coverage computation process
Compute coverage and cumulated coverage
Project the FTS Model
Generate a set of configurations from the FD
VaMoS '15
xavier.devroey@unamur.be 26
Coverage computation process
Compute coverage and cumulated coverage on FTS
Project the FTS Model
Generate a set of configurations from the FD
VaMoS '15
xavier.devroey@unamur.be 27
Results
VaMoS '15
xavier.devroey@unamur.be 28
Results
VaMoS '15
xavier.devroey@unamur.be 29
Results
VaMoS '15
xavier.devroey@unamur.be 30
Observations
• Relatively few configurations are needed to cover 100% of states, transitions and actions– Tendency is confirmed both on academic and Claroline
• T-Wise coverage (SPLCAT) provides better coverage (for all our behavioural criteria) for T = 3
VaMoS '15
xavier.devroey@unamur.be 31
Observations
• For both approaches, there is a need for prioritization and optimal behavioural coverage – E.g., from the 73 config. Generated for Claroline, only 1 is
needed to achieve state coverage
• Threat: Considered FDs have few features and cross-tree constraints
VaMoS '15
CONCLUSION
… and Future Works
xavier.devroey@unamur.be 33
Conclusion
• For our models (9 up to 44 features), a small set of configurations allows good coverage– From 3 to achieve 100% actions and states coverage– Between 5 and 8 for 100% transitions coverage
VaMoS '15
xavier.devroey@unamur.be 34
Conclusion
• For our models (9 up to 44 features), a small set of configurations allows good coverage– From 3 to achieve 100% actions and states coverage– Between 5 and 8 for 100% transitions coverage
• Maximising Behavioural coverage cannot be done only using structural coverage criteria
VaMoS '15
xavier.devroey@unamur.be 35
Future Works
• Formally establish the relationship between structural and behavioural SPL Coverage
VaMoS '15
xavier.devroey@unamur.be 36
Future Works
• Formally establish the relationship between structural and behavioural SPL Coverage
• … in order to define hybrid behavioural/structural criteria
VaMoS '15
xavier.devroey@unamur.be 37
Future Works
• Formally establish the relationship between structural and behavioural SPL Coverage
• … in order to define hybrid behavioural/structural criteria
• Guided by replication of the experiment on more complex case studies (more features, more constraints, more complex behaviour, etc.)
VaMoS '15
THANK YOU FOR YOUR ATTENTION !
Models and tools available at https://projects.info.unamur.be/vibes/