Evolutionary Methods for State-based...
Transcript of Evolutionary Methods for State-based...
![Page 1: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/1.jpg)
Evolutionary Methods for
State-based Testing
PhD Student Raluca LefticaruSupervised by Florentin Ipate
University of Piteşti, Romania
Department of Computer Science
![Page 2: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/2.jpg)
2
Outline
� Motivation
� Search-based software engineering
� Metaheuristic search techniques
� Test data generation for state-based testing
� Experiments and results obtained
� Conclusions
� Questions
![Page 3: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/3.jpg)
3
Motivation
� A lot of research has been done in the field of state-based testing:
� test selection methods for FSMs – the W-method
� coverage criteria for state machines diagrams (like all transitions, full predicate, transition pair, complete sequence, disjunct coverage)
� The automatic generation of test cases from extended state machines (state machines diagrams, X-machines, etc.) is not straightforward.
� For example: which input values to choose for a sequence of methods calls, representing a path in a state machine diagram?
� Idea: Why not to use the power of evolutionary algorithms or other metaheuristics to solve state-based testing problems?
� This answer is given by new field of Search-based Software Engineering
![Page 4: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/4.jpg)
4
Search-based Software
Engineering (SBSE)� Search-based software engineering (SBSE) is a relatively
new approach to transform the software engineering problems into optimization problems, which can be further solved by applying metaheuristic search techniques.
� The term of SBSE was first used by Harman and Jones (2001), although there are some previous papers on this topic.
� Repository of publications on SBSE: http://www.sebase.org/sbse/publications/repository.html
� In 2008 there were published more than 100 papers on SBSE.
� Important events:� The International Workshop on Search-Based Software
Testing, held in conjunction with ICST: 2008 Lillehammer, 2009 Denver
� The International Symposium on Search Based Software Engineering, 2009 Windsor
![Page 5: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/5.jpg)
5
Search-based software
engineering (SBSE)
� Main idea: Transform the software engineering problems into optimization problems, which can be further solved by applying metaheuristics.
� Search techniques: genetic algorithms, hill climbing, alternating variable method, simulated annealing, genetic programming, particle swarm optimization, artificial immune systems, tabu search etc.
� Applications: software testing, requirements engineering, automated maintenance, service-oriented software engineering, compiler optimization, quality assessment, project planning and cost estimation.
![Page 6: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/6.jpg)
6
Number of publications in
SBSE, 1976 - 2009
http://www.sebase.org/sbse/publications/
![Page 7: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/7.jpg)
7
Ratio of SE research fields
involved in SBSE
http://www.sebase.org/sbse/publications/
![Page 8: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/8.jpg)
8
Search-based Software
Testing (SBST)
� Characterized by the usage of guided search techniques for test generation
� The test aim (cover all branches, obtain the WCET) is first transformed into an optimization problem with respect to some fitness (cost or objective) function.
� Search space = the input domain of the test object (program, function).
� The search spaces obtained are usually complex, discontinuous, and non-linear, due to the non-linearity of software
� Therefore metaheuristic search methods are recommended.
![Page 9: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/9.jpg)
9
Search-based Software
Testing (SBST)
� Structural testing: automatically generate input data for programs written mainly in a procedural paradigm.
� Program = directed graph
� Cover the desired graph elements (nodes, branches or paths)
� Functional testing: from Z specification, conformance testing, automatic parking system
� Testing of grey-box properties, for example safety constraints
� Testing non-functional properties, such as worst-case execution time
![Page 10: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/10.jpg)
10
Metaheuristic search
techniques
� Simulated annealing (SA): a random generated ”neighbour”replaces the current solution if it has a better objective value;
otherwise, with the probability p = exp ( -δ / t ).
� Genetic algorithms (GAs): a class of evolutionary algorithms, that use selection, recombination (crossover) and mutation, applied on a population of potential solutions, called chromosomes (or individuals)
� Particle swarm optimization (PS): a population of random solutions, called particles, which maintain their current position, velocity and best position explored so far, fly through the problem space by following the current optimum particles
![Page 11: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/11.jpg)
11
Test data generation for
state-based testing
� Given: some paths in the state machine (obtained according to a certain coverage criteria)
� Use metaheuristic search techniques to find for each method sequence the input values for the parameters, which satisfy the corresponding guards (pre-conditions)
� Questions:
� Encoding: x = (x1, x2, … , xn)
� Fitness function
� Search technique
![Page 12: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/12.jpg)
12
Fitness function calculation
]1,0[__
}1,,1,0{_
___
∈
−∈
+=
levelbranchnormalized
mlevelapproach
levelbranchnormalizedlevelapproachfitness
K
![Page 13: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/13.jpg)
13
Tracey’s objective functions
min(obj(a), obj(b))a \/ b
obj(a) + obj(b)a /\ b
if TRUE then 0 else KBoolean
if b − a ≤ 0 then 0 else (b − a) + Ka ≥ b
if b − a < 0 then 0 else (b − a) + Ka > b
if a − b ≤ 0 then 0 else (a − b) + Ka ≤ b
if a − b < 0 then 0 else (a − b) + Ka < b
if abs(a − b) <> 0 then 0 else Ka ≠ b
if abs(a − b) = 0 then 0
else abs(a − b) + K
a = b
Objective function objPredicate
![Page 14: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/14.jpg)
14
State machine diagram of a
Book class
rese
rve
cancel
rese
rve
can
ce
l
bor(int x
) [ x
= g
etResC
ustId
() ]
res(int x)
[ x >
0 a
nd
x <
> g
etB
orr
CustId()
]
can(int x)
[ x
= g
etR
esC
ustId()
]
res(int x)
[ x >
0 ]
can(int x)
[ x
= g
etR
esC
ustId()
]
Book()
![Page 15: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/15.jpg)
15
Fitness function landscape
BRAxborIdgetResCustxxresx
)( ()][)( ]0[ 2211
→ →=>
ddnorm
norm
K
x x x
xx
−
−=
→
=
=>
−∈
05.11)(
)1,0[]101,0[:
1constant sTracey'
and 0
]50,50[,
121
21
![Page 16: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/16.jpg)
16
� Generating numerical input values for given paths in the state machine, with the approach level + normalized branch levelfitness function
� GAs had a good convergence rate (max. allowed generations 100)
� Heuristic real value crossover inspired from Michalewicz: zi = α・(xi − yi) + xi , 0 < α <1, where x = (x1, . . . , xn), y = (y1, ..., yn), xfitter than y.
� Conformance testing: experiments that used mutation testing and a fitness function of form pre-condition /\ ¬ post-condition.
� Discovered 72-82% of the mutants in 100 generations. After increasing the maximum allowed evolutions to 200 the rest of unequivalent mutants were detected also.
R. Lefticaru, F. Ipate, "Automatic State-Based Test Generation Using Genetic Algorithms“, SYNASC2007
Experiments employing
genetic algorithms
![Page 17: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/17.jpg)
17
Average number of generations, related to:
solutions number / possible solutions number
0
20
40
60
80
100
3.12
E-1
43.
11E
-11
6.25
E-1
01.
25E
-09
5.16
E-0
81.
25E
-07
6.09
E-0
61.
85E
-05
2.46
E-0
43.
47E
-03
2.04
E-0
23.
07E
-01
Generations
![Page 18: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/18.jpg)
18
Average number of failures, related to: solutions
number / possible solutions number
0%
2%
4%
6%
8%
3.12
E-1
43.
11E
-11
6.25
E-1
01.
25E
-09
5.16
E-0
81.
25E
-07
6.09
E-0
61.
85E
-05
2.46
E-0
43.
47E
-03
2.04
E-0
23.
07E
-01
Failures
![Page 19: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/19.jpg)
19
Experiments employing GAs,
PSO, SA
� Aim: generate input values for different state machine paths
� Search techniques: GAs, SA, PSO
� The corresponding fitness landscapes have different complexity: simple (only one minimum), complex (many local minima).
R. Lefticaru, F. Ipate, "Functional Search-based Testing from State Machines", ICST2008
![Page 20: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/20.jpg)
20
Simple objects (one minimum)
� GAs (with heuristic crossover) achieved better results (9 out of 9 cases) than PSO and SA
Function evaluations for simple objects
0
1000
2000
3000
4000
1 2 3 4 5 6 7 8 9
Simple objects
PS
GA
SA
Success rates for simple objects
0%
25%
50%
75%
100%
1 2 3 4 5 6 7 8 9
Simple objects
PS
GA
SA
![Page 21: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/21.jpg)
21
Complex objects (more local
minima)� For more complex landscapes, having more local minima, PSO
outperformed GAs and the difference was statistically significant for 17 out of 21 test objects (confidence 95%)
Success rates for complex objects
0%
25%
50%
75%
100%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Complex objects
PS
GA
![Page 22: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/22.jpg)
22
Measures to Characterize
Search Problems
� Used to characterize the search landscapes, to predict or explain the behavior of search algorithms, to guide the implementation choices to be made.
� Intuitively, some fitness functions might present plateaux along which they provide no guidance.
� Diameter: maximal distance between two points in S, in terms of successive applications of a neighbourhood operator N.
� Autocorrelation: measures the variation of fitness for points that are at the same distance. It characterizes the ruggedness of a landscape: smooth (the neighbours have nearly the same fitness value) or rugged (the fitness values are dissimilar).
� Fitness Distance Correlation (FDC): joint variation of distances and costs.
![Page 23: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/23.jpg)
23
Analysing the fitness
landscape
� The general fitness functions approach level + normalized branch level (al + nbl) may produce results
comparable to those produced by fitness functions, designed especially for a particular situation.
� The best results were obtained when using the al + nbl
functions with GA and PSO.
R. Lefticaru, F. Ipate, "Search-based Testing using State-based Fitness ",
ICSTW 2008 (SBST 2008)
R. Lefticaru, F. Ipate, "A Comparative Landscape Analysis of Fitness
Functions for Search-based Testing", SYNASC 2008
![Page 24: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/24.jpg)
24
Fitness-distance scatterplots
0 1 2 3 4
x 105
0
500
1000
1500
Distance
Fitn
ess
cost2
0 1 2 3 4
x 105
0
1
2
3
4x 10
5
DistanceF
itne
ss
cost3
0 1 2 3 4
x 105
0
1
2
3
4x 10
5
Distance
Fitn
ess
cost4
0 1 2 3 4
x 105
0
0.5
1
1.5
2
2.5
3
Distance
Fitn
ess
cost_ymd
0 1 2 3 4
x 105
0
0.5
1
1.5
2
2.5
Distance
Fitn
ess
cost_dmy
0 1 2 3 4
x 105
1
1.5
2
2.5
Distance
Fitness
cost_dym
![Page 25: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/25.jpg)
25
Conclusions
� The test data obtained with the al+nbl functions can cover difficult paths in the machine.
� A slightly different design (pre-condition /\ ¬ post-condition) of the fitness function can be used for specification conformance testing.
� For more complex landscapes, having more local minima, PSO outperformed GAs.
� For simpler function, having only one minimum GAs achieved better results.
� Hybridizing SA or GAs with local search techniques improved their effectiveness.
� The general fitness functions al + nbl may produce results comparable to those produced by fitness functions, designed especially for a particular situation.
![Page 26: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/26.jpg)
26
Future work
� Analyzing, on a larger benchmark of classes, the effectiveness and the efficiency of several search techniques.
� Finding categories of problems for which certain search algorithms achieve better results.
� Extending the strategy presented for method parameters with morecomplex types and multi-class testing.
� Analyzing other variants of fitness functions.
� Derivation of the fitness function from hierarchical and concurrent state machine diagrams.
![Page 27: Evolutionary Methods for State-based Testingantares.sip.ucm.es/tarot09/index_files/Lefticaru-TAROT09.pdf · sequence, disjunct coverage) The automatic generation of test cases from](https://reader034.fdocuments.net/reader034/viewer/2022042212/5eb575473edc0459a63ccce4/html5/thumbnails/27.jpg)
27
Question & Answers
Thank you for your attention !