[IEEE 2010 22nd International Conference on Tools with Artificial Intelligence (ICTAI) - Arras,...

6
A Multi-Objective Genetic Algorithm to Test Data Generation Gustavo H.L. Pinto, Silvia R. Vergilio Federal University of Paraná Computer Science Department CP: 19081, Centro Politecnico CEP: 81531-970 Jardim das Américas - Curitiba, Brazil [email protected], [email protected]. Abstract Evolutionary testing has successfully applied search based optimization algorithms to the test data generation problem. The existing works use different techniques and fitness functions. However, the used functions consider only one objective, which is, in general, related to the cover- age of a testing criterion. But, in practice, there are many factors that can influence the generation of test data, such as memory consumption, execution time, revealed faults, and etc. Considering this fact, this work explores a multi- objective optimization approach for test data generation. A framework that implements a multi-objective genetic al- gorithm is described. Two different representations for the population are used, which allows the test of procedural and object-oriented code. Combinations of three objectives are experimentally evaluated: coverage of structural test crite- ria, ability to reveal faults, and execution time. 1 Introduction The test data generation to satisfy a test criterion is as- sociated to several limitations, and it can not be completely automated. Because of this, this task usually consumes a lot of effort. Given a criterion C, there is no algorithm to deter- mine whether a test set T is C-adequate, that is, whether T satisfies C, covering all the elements required by C. There is no algorithm even to determine whether such set exists [10]. Due to this fact, the task of test data generation is an open research question. Approaches based on search based algo- rithms, such as Genetic Algorithms (GA), present promis- ing results, and were grouped in an area, called Evolu- tionary Testing [28]. The works on this subject differ on the search based technique and on the fitness func- tion used. Most of them are coverage oriented, address- ing structural and fault based criteria applied to procedu- ral code [15, 18]. The context of object-oriented programs is a more recent research field with a reduced number of works [22, 23, 24, 27]. A limitation of these works is that they formulate the test generation problem as a single objective search problem, which is, in general, the coverage of a path (or required ele- ment). However, there are diverse characteristics desired for the test data that should be considered to guide the genera- tion task, such as: ability to reveal certain kind of faults, re- duced execution time and memory consumption, and other factors associated to the development environment that can be related to the type of software being developed. In this way, we notice that the test data generation prob- lem is in fact multi-objective, since it can depend on multi- ple variables (objectives). For such problems there is no sin- gle solution that satisfies all the objectives. This is because the objectives can be in conflict, for example, coverage and execution time. Considering this fact, in this paper, we explore a new multi-objective approach for test data generation. A frame- work is described that uses two representations for the test data to allow the test of procedural and object-oriented code, and the integration with the testing tools: Poketool [17], Proteum [8] and JaBUTi [26]. The framework imple- ments the NSGA-II (Non-dominated Sorting Genetic Algo- rithm) [7], an algorithm that evaluates the solutions accord- ing to Pareto dominance concepts [20] considering different objectives: coverage of a chosen structural criterion, execu- tion time, and ability to reveal faults. Some experimental results of three case studies show that the implemented al- gorithm produces a set of solutions that represent a good trade off between the objectives to be satisfied by the test data generated. The rest of the paper is organized as follows. Section 2 introduces the framework and implemented algorithm. Sec- tion 3 presents experimental evaluation results. Finally, in Section 4 we conclude the paper and present the main con- tributions of the work. 2010 22nd International Conference on Tools with Artificial Intelligence 1082-3409/10 $26.00 © 2010 IEEE DOI 10.1109/ICTAI.2010.26 129

Transcript of [IEEE 2010 22nd International Conference on Tools with Artificial Intelligence (ICTAI) - Arras,...

Page 1: [IEEE 2010 22nd International Conference on Tools with Artificial Intelligence (ICTAI) - Arras, France (2010.10.27-2010.10.29)] 2010 22nd IEEE International Conference on Tools with

A Multi-Objective Genetic Algorithm to Test Data Generation

Gustavo H.L. Pinto, Silvia R. VergilioFederal University of Paraná

Computer Science Department

CP: 19081, Centro Politecnico CEP: 81531-970

Jardim das Américas - Curitiba, Brazil

[email protected], [email protected].

Abstract

Evolutionary testing has successfully applied searchbased optimization algorithms to the test data generationproblem. The existing works use different techniques andfitness functions. However, the used functions consider onlyone objective, which is, in general, related to the cover-age of a testing criterion. But, in practice, there are manyfactors that can influence the generation of test data, suchas memory consumption, execution time, revealed faults,and etc. Considering this fact, this work explores a multi-objective optimization approach for test data generation.A framework that implements a multi-objective genetic al-gorithm is described. Two different representations for thepopulation are used, which allows the test of procedural andobject-oriented code. Combinations of three objectives areexperimentally evaluated: coverage of structural test crite-ria, ability to reveal faults, and execution time.

1 Introduction

The test data generation to satisfy a test criterion is as-sociated to several limitations, and it can not be completelyautomated. Because of this, this task usually consumes a lotof effort. Given a criterion C, there is no algorithm to deter-mine whether a test set T is C-adequate, that is, whether Tsatisfies C, covering all the elements required by C. There isno algorithm even to determine whether such set exists [10].

Due to this fact, the task of test data generation is an openresearch question. Approaches based on search based algo-rithms, such as Genetic Algorithms (GA), present promis-ing results, and were grouped in an area, called Evolu-tionary Testing [28]. The works on this subject differon the search based technique and on the fitness func-tion used. Most of them are coverage oriented, address-ing structural and fault based criteria applied to procedu-ral code [15, 18]. The context of object-oriented programs

is a more recent research field with a reduced number ofworks [22, 23, 24, 27].

A limitation of these works is that they formulate the testgeneration problem as a single objective search problem,which is, in general, the coverage of a path (or required ele-ment). However, there are diverse characteristics desired forthe test data that should be considered to guide the genera-tion task, such as: ability to reveal certain kind of faults, re-duced execution time and memory consumption, and otherfactors associated to the development environment that canbe related to the type of software being developed.

In this way, we notice that the test data generation prob-lem is in fact multi-objective, since it can depend on multi-ple variables (objectives). For such problems there is no sin-gle solution that satisfies all the objectives. This is becausethe objectives can be in conflict, for example, coverage andexecution time.

Considering this fact, in this paper, we explore a newmulti-objective approach for test data generation. A frame-work is described that uses two representations for the testdata to allow the test of procedural and object-oriented code,and the integration with the testing tools: Poketool [17],Proteum [8] and JaBUTi [26]. The framework imple-ments the NSGA-II (Non-dominated Sorting Genetic Algo-rithm) [7], an algorithm that evaluates the solutions accord-ing to Pareto dominance concepts [20] considering differentobjectives: coverage of a chosen structural criterion, execu-tion time, and ability to reveal faults. Some experimentalresults of three case studies show that the implemented al-gorithm produces a set of solutions that represent a goodtrade off between the objectives to be satisfied by the testdata generated.

The rest of the paper is organized as follows. Section 2introduces the framework and implemented algorithm. Sec-tion 3 presents experimental evaluation results. Finally, inSection 4 we conclude the paper and present the main con-tributions of the work.

2010 22nd International Conference on Tools with Artificial Intelligence

1082-3409/10 $26.00 © 2010 IEEE

DOI 10.1109/ICTAI.2010.26

129

Page 2: [IEEE 2010 22nd International Conference on Tools with Artificial Intelligence (ICTAI) - Arras, France (2010.10.27-2010.10.29)] 2010 22nd IEEE International Conference on Tools with

2 Test Data Generation by a Multi-objectiveApproach

The framework has three modules: i) brainModule, ii)testingModule and iii) configModule. The communicationamong the modules is illustrated in Figure 1. The parame-ters provided by the tester are received (‘a’), and format-ted by the configModule (‘b’). The brainModule imple-ments the NSGA-II algorithm, a modification of the orig-inal NSGA proposed by Deb [7], and is responsible by theevolution process. The test data sets are passed to the test-ingModule (‘c’), which integrates the testing tools and isresponsible for the transformation of the data according tothe format used by the tools. After the the test data execu-tion, the testingModule returns information about the cov-erage produced by the tools (‘d’) to allow the evolution ofthe population.

Figure 1. Modules of the framework

The configModule needs the following parameters: a) in-formation related to the evolution process: maximum num-ber of generations or maximum execution time; crossover,mutation and relation rates (related to the application of thegenetic operators); number of individuals in the population;b) information to generate the population: number of testdata in each individual, type of the input variables or num-ber maximum of methods calls; c) objectives to be used inthe fitness evaluation; d) information about the test tools:language and chosen test criterion.

The brainModule is responsible for the: initial genera-tion of the individuals that are test data sets, evolution ofthe population by the application of the genetic operatorsand fitness evaluation. It implements the multi-objective al-gorithm. Important aspects of any meta-heuristic algorithmare: the representation of the population and the choice ofthe fitness function, which are presented next.

2.1 Representation of the population

Works on search based test, in general, represents theinput data as an individual of the population. In this workthe individual represents a set of test data to be evolved. Themaximum size of such set is determined by the tester. This

structure is presented in Figure 2, in which the individual iscomposed by n test data. With this representation, differentsolutions can be explored in the search space.

Figure 2. An individual in the population

Two possible representations for the test data in each in-dividual can be used, depending on the chosen language andtool, representing test data for either procedural or object-oriented programs. In the procedural context, the input con-sists in a sequence of values to execute the program. Thetype and number of the values are fixed and provided by testand vary according to the program being tested. Because ofthis, the codification of the input is simple.

In the context of object-oriented code, however, the testdata is not a simple sequence of values. It contains se-quences of invocations to constructors and methods, in-cluding the correspondent parameters. To represent eachtest data, the framework used the following grammar, intro-duced by Tonella [24]:

<chromosome> ::= <actions> @ <values><actions> ::= <action> {: <actions>}?<action> ::= $id = constructor ( {<parameters>}? )

| $id = class # null| $id . method ( {<parameters>? )

<parameters> ::= <parameter> {, <parameters>}?<parameter> ::= builtin-type {<generator>}?

| $id<generator> ::= [ low ; up ]

| [ genClass ]<values> ::= <value> {, <values>}?<value> ::= integer

| real| boolean| string

The test data is composed by two parts, separated by thecharacter ‘@’. The first part, the non-terminal <actions>contains a sequence of constructors and methods invoca-tions, separated by ‘:’.

Each <action> can either construct a new object, as-signed to a variable $id, or result in a method invocation.A special case involves the value null, that corresponds toa reference for a non existing object (if no constructor forthis object is invoked). The value null is preceded by theclass of $id (separated by the character ‘#’) and can be usedas a parameter only if its type matches to that of the formalparameter. Parameters of the methods and constructors areprimitive types of the language such as int, real, booleanor a variable $id. The second part of the chromosome con-tains input values used in each invocation, separated by thecharacter ‘,’.

130

Page 3: [IEEE 2010 22nd International Conference on Tools with Artificial Intelligence (ICTAI) - Arras, France (2010.10.27-2010.10.29)] 2010 22nd IEEE International Conference on Tools with

2.2 Genetic operators

The genetic operators transform the population throughsuccessive generations and help to maintain diversity andadaptation characteristics obtained by previous generations.In the context of procedural code, the operators are ap-plied by considering the type of each provided input. In thecrossover, if the values are integer they can be summed orsubtracted to compose another test data. If they are string,they can be concatenated. The mutation occurs accordingto a probability. In such case, a new value is generated con-sidered the type of the input value. Duplicate test data areeliminated.

In the context of object-oriented programs, the operatorsare applied as follows. The mutation operator changes, in-serts or removes inputs, constructors and methods invoca-tions. The crossover operator selects points that can be inthe sequence of methods invocations or in the part that con-tains the parameters.

The operators of relation change test data between indi-viduals to ensure diversity. They apply: half relation andrandom relation. Consider the initial sets A = {a, b, c, d, e,f} and B = {g, h, i, j, k, l}. The operator of half relationderives two new sets. The first one containing the first halfof A and the second half of B. The second one containingthe remaining parts of A and B. The following sets are gen-erated: HRi = {a, b, c, j, k, l} and HRii = {g, h, i, d, e, f}.Otherwise, the operator Random relation randomly selectsindex for the elements that will be changed.

2.3 Fitness Evaluation

Each individual can be evaluated according two or moreobjectives chosen by the tester. In this version of the frame-work, three functions are available.

- Coverage of a structural criterion: the tester can choosea criterion implemented by the integrated tools. The firstone, Poketool, supports the test of C programs with thecontrol-flow and Potential Uses criteria family [17]. JaBUTisupports all-nodes, all-edges, all-uses and all-potential uses,for the test of Java programs [26] and derives the requiredelements directly from the Java bytecode. The coverage ofeach set x (individual) to be maximized is given by:

Fitnessx =nr. elements coveredx * 100

total number of required elements

- Execution time: the execution time of the test data setx is given by the sum of the execution time of each test dataon x.

- Ability to reveal faults: to evaluate this ability for a testdata set x it was used mutation test and the essential opera-tors [2] implemented by the tool Proteum [8]. If the test data

is capable to kill the mutants generated by a mutation oper-ator, we consider that this data test is capable to reveal thefault described by the corresponding operator. The abilityis given by the mutation score of x provided by Proteum.

2.4 Evolution Process

The evolution process follows these steps. In the initialstep, the test data are generated by using the chosen repre-sentation, according to the language. The individuals in thepopulation are test data sets. The initial population is ran-domly generated. A loop starts the evolution process. Thefitness of each individual is given by the tools, by evaluatingeach test data in the set. After this, the dominated and non-dominated solutions are identified according to the Paretodominance criterion. Then, the criterion given by the mul-tiple objectives is applied and the selected individuals areused in other evolution, the evolution of the test data.

In such process, the selection based on a roulette is usedto select the test data in the individual that will suffer the ac-tion of the genetic operators (mutation and crossover). Thismeans that the test data are also evolved, and it is supposedthat the individuals (the test data sets) will save the best testdata trough the iterations. At the end of an iteration, the testdata are changed among individuals by the relation opera-tors, to maintain diversity. This is related to the evolution ofthe test data sets.

Finally, the set of the best solutions are added to the newpopulation, and are returned as output by the algorithm. Theevolution continues until the stop criterion is reached, thatis, a number maximum of generations (or the execution timedetermined by the tester) is reached. After the loop, the in-dividuals are analysed and the non-dominated solutions areadded to the set of non-dominated solutions found duringthe role process.

3 Evaluation Studies

The implemented approach was evaluated in three casestudies. They were chosen to make possible an evaluationin both test contexts (procedural and object-oriented code).Other points to be explored by the case studies were: theuse of different test criteria (based on control and data flow)and the combination of the three objectives implemented bythe framework to guide the test data generation.

The configuration was empirically adjusted. In each casestudy, the algorithm was executed with variations in thenumber of generations, number of individuals in the popu-lation, number of test data, and etc. The rates related to thegenetic operators were fixed following suggestions foundin the literature [9], and after this, adjusted. They are: 0.8for crossover rate; 0.2 for mutation; and 0.7 for the relationoperator. Chosen the configuration, the results obtained by

131

Page 4: [IEEE 2010 22nd International Conference on Tools with Artificial Intelligence (ICTAI) - Arras, France (2010.10.27-2010.10.29)] 2010 22nd IEEE International Conference on Tools with

NSGA-II were compared to a random strategy (RS) by con-figuring the number of generations equal to 1.

In both strategies, the algorithms were executed 10times. After all the executions, to allow a visual com-parison, a front was obtained composed only by the non-dominated solutions, considering the solutions of all execu-tions. In many cases, mainly considering RS, after this steponly one solution was obtained. This means that this solu-tion dominates the solutions found in all executions. Next,the solutions found and the configuration used for each casestudy are presented.

3.1 Case Study 1

This study used two object-oriented programs and twoobjectives: coverage of a structural criterion and executiontime. The programs are: Find.java and TriTyp.java1. Asmentioned in Section 3 the program TriTyp.Java determinesif three integer numbers form a triangle and in the affirma-tive case returns the type of the formed triangle. The otherprogram Find.Java searches for an entry in a vector. Thefollowing criteria implemented by JaBUTi were used: all-edgesei (AEi) and all-usesei(USi). The configuration usedis in Table 1. The solutions found for each program andcriterion are in Table 2.

Table 1. Configurations of Case Study 1

Program Criterion numGenerations numIndividuals numTestDataTriTyp AEi 100 100 50TriTyp USi 100 100 50Find AEi 50 50 50Find USi 50 100 50

Table 2. Results of Case Study 1

Program Criterion NSGA-II RSCov. Exec.Time Cov. Exec.Time(ms)(%) (ms) (%) (ms)81 319 65 28476 297

TriTyp AEi 74 28172 27852 344

USi 69 315 42 34940 331

AEi 95 297 88 32793 292

Find 88 303 86 247USi 87 220 76 240

77 200

We can observe, as it was expected, that it is more diffi-cult to satisfy the data-flow based criterion. For programTriTyp and the criterion all-edges, NSGA-II presented 5

1Used in diverse works reported in the literature [4, 21]. Available onhttp://www.infcr.uclm.es/www/mpolo/stvr/.

non-dominated solutions and RS one. The RS solution isdominated by the NSGA solutions. The NSGA solutionsvary from 244 to 319ms and coverage of 52 to 81%. Thealgorithm could select the most relevant points in the searchspace. For the criterion US, NSGA-II presented only onesolution that also dominates both solutions found by RS.Similar result was obtained to program Find. NSGA-II pre-sented a greater number of solutions, which dominate thesolutions found by RS.

3.2 Case Study 2

In this study, two objectives were evaluated: the cover-age of the data-flow based criterion all-potential-uses (PU)(or the coverage of the criterion all-edges (AE)) imple-mented by Poketool, and the ability of reveal faults, de-scribed by the essential operators of Proteum. It was usedthe program compress.c2. The program replaces charactersthat appear more than three times in a string by a sequencethat indicates the number of repetitions, and returns a stringwith a lower number of characters.

In the case, for both criteria all the parameters were setwith 50, except the number of individuals in the populationfor criterion PU, which was set with 100. (see Table 3).

Table 3. Results of Case Study 2

Program Criterion NSGA-II RSCov. Exec.Time Cov. Exec.Time(%) (ms) (%) (ms)

compress AE 100 77 100 70PU 83.33 77 83.33 70

We observe that both strategies reach the same coveragefor both structural criterion. The obtained value for the cov-erage is optimal (or near to the optimal, since the criterionPU requires around 10% of infeasible elements [25]). How-ever, the test set generated by NSGA-II has greater abilityto reveal faults.

3.3 Case Study 3

Again, with the program compress.c, three objectiveswere investigated: the coverage of the criterion all-potential-uses (PU), the ability to reveal faults described bythe essential operators, and execution time. The parametersused are: number of generations and individuals equal to100, and number of test data 10 (see Table 4).

Considering more than two objectives does not implylower efficiency. NSGA-II obtained solutions with the samecoverage reached with two objectives, and at the same timeto minimize the third objective, execution time. The same

2Available on the book of Kernighan [16] and also used by other au-thors in the literature [29].

132

Page 5: [IEEE 2010 22nd International Conference on Tools with Artificial Intelligence (ICTAI) - Arras, France (2010.10.27-2010.10.29)] 2010 22nd IEEE International Conference on Tools with

does not happen with RS. The solutions obtained by NSGAdominate the solution found by RS.

Table 4. Results of Case Study 3

NSGA-II RSCov. Fault Exec.Time Cov. Fault Exec.Time(%) (%) (ms) (%) (%) (ms)

83.33 77 80 78.89 70 8783.33 76 7778.89 77 76

3.4 Analysis

By analysing the results of all case studies we can ob-serve that the solutions found by NSGA-II dominate all thesolutions found by RS, independently of the test criterionand objective being considered. The implemented algo-rithm is capable to improve the quality of the data gener-ated, and offer a set of good solutions to be used accordingto the test goals. To compare the strategies, the indicatorHypervolume [32] was used. This indicator shows how thesolutions found by the algorithms are spread in the searchspace. The values of Hypervolume were compared throughthe Mann-Withney U-test [6]. It is a non-parametric test,used to check the null hypothesis of two samples being sim-ilar. It was adopted a significance level (p-value) of 0.05,and all the samples obtained p-values less than < 0.05, in-dicating the the NSGA-II presents statistically better solu-tions.

The studies were done in a computer Intel(R) Xeon(R)- 7GB, operational system Debian 5.0. We observe that greaternumbers of individuals and test data increase the costs ofboth strategies. It seems that the test criterion used doesnot influence significantly on this cost. We observe thatthe NSGA-II cost (in terms of execution time) is almostthree times greater than the RS cost. So we suggest that thestrategies can be used in a complementary way. NSGA-IIcan be used to improve the random sets, mainly consideringstrength criteria, such as data flow based.

Other observed point, to be investigated in future exper-iments, is that the coverage of the criterion and ability toreveal faults seems to be dependent. Improving coverageimplies to reveal more faults. Hence, a combination thatmakes the use of a multi-objective approach more suitableis that one that includes the execution time.

4 Related Work

In the Software Engineering area, the use of multi-objective optimization is an emergent research area [14]. Ithas been explored to reduce project costs and to configuresoftware project times [1]; to select requirements for soft-ware release [31], to software refactoring and design [3, 12].

In the test activity these algorithms were used for selectionof test cases [30] and also to test data generation. Cao etal. [5] present a method based on similarity of executedpaths to generate test data to cover a specific path. Ghiduk etal. [11] introduce an algorithm to ensure coverage of a data-flow based criterion and to reduce the number of generatedtest data. With the same goal, the work [19] implementsmulti-objective algorithms to maximize coverage of controland data flow based criteria, and to minimize the number oftest data. A work that is very similar to ours is the work ofHarmam et al. [13]. In such work, the authors also used amulti-objective Genetic Algorithm to maximize the cover-age of branches and to reduce consumption of memory.

Our work, differently of the abovementioned ones, is in-tegrated with two test tools, which allows the use of a multi-objective genetic algorithm in two contexts, procedural andobject-oriented programs. The criteria implemented in theobject-oriented context derive their test requirements fromthe Java bytecode. The representation used for the individ-uals allows the test data generation even if the source codeis not available. This is very common in the test of mostcomponents. In addition to this, the the multi-objective ap-proach considers other objectives and factors that can influ-ence the test data generation and are required for the testdata: execution time and kind of faults to be revealed.

5 Conclusions

This work explored a multi-objective approach for testdata generation and described a framework that implementsthe algorithm NSGA-II considering possible combinationsof three objectives: coverage of structural criteria, executiontime, and ability to reveal faults.

The framework implements two kind of representationsfor the population and the NSGA-II algorithm. The firstrepresentation allows the test of C programs and the inte-gration with the tool Poketool that implements the criteriabased on control and data flow. The second representationallows the test of object-oriented code and the integrationwith the tool JaBUTi, which implements control and dataflow based criteria that derive their test requirements di-rectly from the Java bytecode. The framework allows thetest data generation even if the source code is not available.

The use of the implemented algorithm in both contextswas evaluated in three case studies and compared with arandom strategy. The results show that the multi-objectivealgorithm get better solutions and improvements for all theobjectives, and a large variety of solutions to be chosen ac-cording to the testing purposes. The solutions found by therandom strategy are always dominated by the NSGA-II. Theuse of the framework contributes to reduce the tester’s ef-fort. It is justified if there are different characteristics de-sired for the test data to be generated, which are in conflit.

133

Page 6: [IEEE 2010 22nd International Conference on Tools with Artificial Intelligence (ICTAI) - Arras, France (2010.10.27-2010.10.29)] 2010 22nd IEEE International Conference on Tools with

Other research studies include the implementation andevaluation of other objectives in the framework, as well as,other meta-heuristic algorithms. These improvements willallow the conduction of new evaluation experiments.

References

[1] E. Alba and F. Chicano. Software project management withGAs. Information Sciences, 177(11):2380–2401, June 2007.

[2] E. F. Barbosa, J. C. Maldonado, and A. M. R. Vincenzi. To-ward the determination of sufficient mutant operators for C.Soft. Test., Verific. and Reliab., 11(2):113–136, 2001.

[3] M. Bowman, L. C.Briand, and Y. Labiche. Multi-objectivegenetic algorithm to support class responsibility assign-ment. In IEEE International Conference on Software Main-tenance, pages 124–133, October 2007.

[4] L. C. Briand, Y. Labiche, and Z. Bawar. Using ma-chine learning to refine black-box test specifications and testsuites. In The Eighth International Conference on QualitySoftware, pages 135–144, 2008.

[5] Y. Cao, C. Hu, and L. Li. Search-based multi-paths test datageneration for structure-oriented testing. In ACM/SIGEVOSummit on Genetic and Evolutionary Computation, pages25–32, 2009.

[6] W. J. Conover. On methods of handling ties in the wilcoxonsigned-rank test. Journal of the American Statistical Associ-ation, pages 985–988, December 1973.

[7] K. Deb and N. Srinivas. Multiobjective optimization usingnondominated sorting in genetic algorithms. In IEEE Trans.on Evolutionary Computation, pages 221–248, 1994.

[8] M. E. Delamaro and J. C. Maldonado. Proteum - a tool forthe assesment of test adequacy for C programs. In Con-ference on Performability in Computing Systems (PCS 96),pages 79–95, July 1996.

[9] J. D. Farmer, N. Packard, and A. Perelson. Introduction togenetic algorithms. MIT press, 1 edition, 1997.

[10] P. G. Frankl and E. J. Weyuker. An applicable family of dataflow testing criteria. IEEE Transactions Software Engineer-ing, 14(10):1483–1498, 1988.

[11] A. S. Ghiduk, M. J. Harrold, and M. R. Girgis. Using geneticalgorithms to aid test-data generation for data-flow cover-age. In 14th Asia-Pacific Software Engineering Conference(APSEC), pages 41–48, December 2007.

[12] L. Grunske. Identifying "good" architectural design alter-natives with multi-objective optimization strategies. In Pro-ceedings of the 28th International Conference on SoftwareEngineering, pages 849–852, May 2006.

[13] M. Harman, K. Lakhotia, and P. McMinn. A multi-objectiveapproach to search-based test data generation. In Geneticand Evolutionary Computation Conference, 2007.

[14] M. Harman, S. A. Mansouri, and Y. Zhang. Search basedsoftware engineering: A comprehensive analysis and reviewof trends techniques and applications. Technical Report TR-09-03, Department of Computer Science, King’s CollegeLondon, April 2009.

[15] B. F. Jones, H. H. Sthamer, and D. E. Eyres. Automaticstructural testing using genetic algorithms. Software Engi-neering Journal, pages 299–306, 1996.

[16] B. W. Kernighan. The C Programming Language. Prentice-Hall, Englewood Cliffs New Jersey, 1978.

[17] J. C. Maldonado, M. L. Chaim, and M. Jino. Briding thegap in the presence of infeasible paths: Potential uses test-ing criteria. In XII International Conference of the ChileanScience Computer Society, pages 323–340, October 1992.

[18] C. C. Michael, G. McGraw, and M. A. Schatz. Generatingsoftware test data by evolution. IEEE Transactions on Soft-ware Engineering, 27(12):1085–1110, December 2001.

[19] N. Oster and F. Saglietti. Automatic test data generation bymulti-objective optimisation. In SAFECOMP, pages 426–438, 2006.

[20] V. Pareto, editor. Manuel D”Economie Politique. Ams Pr,1927.

[21] M. Polo, M. Piattini, and I. García-Rodríguez. Decreasingthe cost of mutation testing with second-order mutants. Soft-ware Testing Verification Reliability, 19(2):111–131, 2009.

[22] J. C. B. Ribeiro. Search-based test case generation forobject-oriented Java software using strongly-typed geneticprogramming. In Genetic and Evolutionary ComputationConference, 2008.

[23] A. Seesing and H.-G. Gross. A genetic programming ap-proach to automated test generation for object-oriented soft-ware. International Transactions on System Science and Ap-plications, 1(2):127–134, October 2006.

[24] P. Tonella. Evolutionary testing of classes. In ACM SIG-SOFT International Symposium on Software Testing andAnalysis, pages 119–128, 2004.

[25] S. R. Vergilio and M. J. J. C. Maldonado. Infeasible paths inthe context of data flow based testing. Journal of the Brazil-ian Computer Society, 12(1), 2006.

[26] A. M. R. Vincenzi, M. E. Delamaro, J. C. Maldonado, andW. E. Wong. Establishing structural testing criteria for Javabytecode. Software: Practice and Experience, 36(14):1513–1541, 2006.

[27] S. Wappler and F. Lammermann. Using evolutionary al-gorithms for the unit testing of object-oriented software.In Conference on Genetic and Evolutionary Computation(GECCO), pages 1053–1060, 2005.

[28] J. Wegener, A. Baresel, and H. Sthamer. Using evolutionarytesting to improve efficiency and quality in software testing.research and technology. In 2nd Asia-Pacific Conference onSoftware Testing Analysis and Review (AsiaSTAR, 2002.

[29] E. J. Weyuker. More experience with data flow testing.IEEE Transactions on Software Engineering, 19(9):912–919, 1993.

[30] S. Yoo and M. Harman. Pareto efficient multi-objectivetest case selection. In International Symposium on SoftwareTesting and Analysis, 2007.

[31] Y. Zhang, M. Harman, and S. A. Mansouri. The multi-objective next release problem. In Conference on Geneticand Evolutionary Computation (GECCO), pages 1129–1137, 2007.

[32] E. Zitzler, D. Brockhoff, and L. Thiele. The hypervolumeindicator revisited: On the design of pareto-compliant in-dicators via weighted integration. In Conference on Evo-lutionary Multi-Criterion Optimization (EMO 2007), pages862–876, 2006.

134