Covering SPL Behaviour with Sampled Configurations: An Initial Assessment

www.unamur.be

Covering SPL Behaviour with Sampled Configurations: An Initial Assessment

Xavier Devroey <[email protected]> Gilles Perrouin <[email protected]>Axel Legay <[email protected]>Pierre-Yves Schobbens <[email protected]>Patrick Heymans <[email protected]>

9th International Workshop on Variability Modelling of Software-intensive Systems, VaMoS ‘15

Hildesheim, Germany

2

Introduction

• Software Product Line Model Based Testing• Structural approaches– Sample configurations to test in a Feature Diagram– Similarity – Pairwise

• Behavioural approaches– Select test cases in a behavioural model of the SPL

VaMoS '15 [email protected]

Devroey et al. (2014). Towards Statistical Prioritization for Software Product Lines Testing. In VaMoS ’14.

3

Introduction

• Software Product Line Model Based Testing• Structural approaches– Sample configurations to test in a Feature Diagram– Similarity – Pairwise

• Behavioural approaches– Select test cases in a behavioural model of the SPL

Practically, which behavioural coverage similarity and pairwise sampling do achieve ?

VaMoS '15 [email protected]

SPECIFYING A PRODUCT LINE

Featured Transition Systems (FTSs) [Classen et al. 2011]

[email protected] 5VaMoS '15

Images credit: http://hedbonstudios.deviantart.com

[email protected] 6VaMoS '15

Images credit: http://hedbonstudios.deviantart.com

[email protected] 7

Featured Transition System

VaMoS '15

STRUCTURAL VS BEHAVIOURAL COVERAGE CRITERIA

Image credits: http://domonated.deviantart.com

[email protected] 9

Structural Coverage

• Which configurations do I select in the FD ?• Pairwise criterion– All pairs of features appears in at least one configuration to test– T-wise – 3-wise interactions are common [Steffens et al. 2012]

VaMoS '15

[email protected] 10

Structural Coverage

• Which configurations do I select in the FD ?• Pairwise criteria– All pairs of features appears in at least one configuration to test– T-wise – 3-wise interactions are common [Steffens et al. 2012]

• (Di)Similarity criteria– Dissimilar configurations to test– With few selected/deselected features in common– Uses a distance function (easy to compute)– Empirically validated [Henard et al. 2013]

VaMoS '15

Steffens et al. (2012). Industrial Evaluation of Pairwise SPL Testing with MoSo-PoLiTe. In VaMoS ‘12


Structural Coverage

VaMoS '15


Behavioural Coverage

• Behavioural test case – Sequence of actions in a FTS– Executable by a valid product of the SPL

VaMoS '15

Devroey et al. (2014). Coverage Criteria for Behavioural Testing of Software Product Lines. In ISoLA’1



VaMoS '15



• Behavioural test case – Sequence of actions in a FTS– Executable by a valid product of the SPL

• Coverage criteria– Executable set of test cases, FTS → [0, 1]– State coverage– Transition coverage – Transition-pairs coverage

VaMoS '15




Exec. Test-Suite TS is Minimal for cov iff

• There is no exec. TS’ with a better coverage cov

• There is no exec. TS’ with fewer Test-Cases

Exec. Test-Suite TS is P-Minimal for cov iff

• There is no exec. TS’ with a better coverage cov

• There is no exec. TS’ which request fewer products to execute TS’

SPLat @ SPLC'14



FTS Test-Case Selection

Number of Exec T.

C.

Number of

Products

Test Case Selection

Coverage

SPLat @ SPLC'14



VaMoS '15

EXPERIMENT

… using PLEDGE and SPLCAT


Setup

• SPLCAT– T-wise criterion, with T in [1,3]

• PLEDGE– (Di)Similarity criterion– Uses a genetic algorithm

• Number of configurations (population) given by SPLCAT• Time = 60 sec. (default time)

VaMoS '15


Coverage computation process

Compute coverage and cumulated coverage

Projects the FTS Model

Generate a set of configurations from the FD

VaMoS '15



Compute coverage and cumulated coverage

Project the FTS Model


VaMoS '15



Compute coverage and cumulated coverage on FTS

Project the FTS Model


VaMoS '15


Results

VaMoS '15


Observations

• Relatively few configurations are needed to cover 100% of states, transitions and actions– Tendency is confirmed both on academic and Claroline

• T-Wise coverage (SPLCAT) provides better coverage (for all our behavioural criteria) for T = 3

VaMoS '15


Observations

• For both approaches, there is a need for prioritization and optimal behavioural coverage – E.g., from the 73 config. Generated for Claroline, only 1 is

needed to achieve state coverage

• Threat: Considered FDs have few features and cross-tree constraints

VaMoS '15

CONCLUSION

… and Future Works


Conclusion

• For our models (9 up to 44 features), a small set of configurations allows good coverage– From 3 to achieve 100% actions and states coverage– Between 5 and 8 for 100% transitions coverage

VaMoS '15


Conclusion

• For our models (9 up to 44 features), a small set of configurations allows good coverage– From 3 to achieve 100% actions and states coverage– Between 5 and 8 for 100% transitions coverage

• Maximising Behavioural coverage cannot be done only using structural coverage criteria

VaMoS '15


Future Works

• Formally establish the relationship between structural and behavioural SPL Coverage

VaMoS '15


Future Works


• … in order to define hybrid behavioural/structural criteria

VaMoS '15


Future Works


• … in order to define hybrid behavioural/structural criteria

• Guided by replication of the experiment on more complex case studies (more features, more constraints, more complex behaviour, etc.)

VaMoS '15

THANK YOU FOR YOUR ATTENTION !

Models and tools available at https://projects.info.unamur.be/vibes/

Covering SPL Behaviour with Sampled Configurations: An Initial Assessment

Science

Transcript of Covering SPL Behaviour with Sampled Configurations: An Initial Assessment