Software Testing and Validation SWE 434 Mutation testing.
-
Upload
ashlie-thompson -
Category
Documents
-
view
224 -
download
3
Transcript of Software Testing and Validation SWE 434 Mutation testing.
Software Testing and Validation
SWE 434Mutation testing
2
Dynamic Testing• Selection of test data is broadly based on the following
techniques:
– Functional program testing
– Control flow testing
– Data flow testing
• Questions:
– How to assess the effectiveness of the test suite ?
– How many bugs are remaining ?
3
Mutation testing• Create a number of mutants, i.e., faulty versions of program
– Each mutant contains one fault– Fault created by using mutant operators
• Run test on the mutants (random or selected)– When a test case reveals a fault, save test case and remove mutant
from the set, i.e., it is killed– Continue until all mutants are killed
• Results in a set of test cases with high quality• Need for automation
4
Mutation Testing
• Modify a program by introducing a single small change to the code
• A modified program is called mutant
• A mutant is said to be killed when the execution of test case cause it to fail. The mutant is considered to be deadGiven a mutant m M for a ground string program P and a test t, t is said to kill m if and only if the output of t on P is different from the output of t on m.
Mutation Testing
• Mutants
int getMax(int x, int y) {int getMax(int x, int y) { int max; int max;
if (x >y) if (x >=y) max = x; max = x;
else else max = y; max = y; return max; return max;} }
5
int getMax(int x, int y) { int max;
if (x >y) max = x;
else max = x;return max;}
Examples of Mutant Operators
6
Mutant operator In program In mutant
Variable replacement z=x*y+1; x=x*y+1;z=x*x+1;
Relational operator replacement
if (x<y) if(x>y)if(x<=y)
Off-by-1 z=x*y+1; z=x*(y+1)+1;z=(x+1)*y+1;
Replacement by 0 z=x*y+1; z=0*y+1;z=0;
Arithmetic operator replacement
z=x*y+1; z=x*y-1;z=x+y-1;
7
Mutation Testing
• A mutant is an equivalent tot the given program if it always produce the same output as the original program
• A mutant is called killable or stubborn, if the existing set of test cases is insufficient to kill it
• A mutation score for a set of test cases is the percentage of non-equivalent mutants killed by the test suite
• The test suite is said to be mutation-adequate if its mutation score is 100%
(c) 2007 Mauro Pezzè & Michal Young Ch 16, slide 8
How mutants survive
• A mutant may be equivalent to the original program– Maybe changing (x < 0) to (x <= 0) didn’t change the
output at all! The seeded “fault” is not really a “fault”.• Determining whether a mutant is equivalent may be easy or
hard; in the worst case it is undecideable
• Or the test suite could be inadequate– If the mutant could have been killed, but was not, it
indicates a weakness in the test suite– But adding a test case for just this mutant is a bad idea.
We care about the real bugs, not the fakes!
9
Mutation Testing: an example
Consider the following program P:
• main(argc,argv)• int argc, r, i;• char *argv[];• { r = 1;• for i = 2 to 3 do• if (atoi(argv[i-1]) > atoi(argv[r-1])) r = i;• printf(“Value of the rank is %d \n”, r);• exit(0); }
Test Case 1: input: 1 2 3 output: Value of the rank is 3Test Case 2: input: 1 2 1 output: Values of the rank is 2Test Case 3: input: 3 1 2 output: Value of the rank is 1
10
Mutation Testing: an example
Test cases for the program P:
• Test Case 1:
input: 1 2 3
output: Value of the rank is 3• Test Case 2:
input: 1 2 1
output: Values of the rank is 2• Test Case 3:
input: 3 1 2
output: Value of the rank is 1
11
Mutation Testing: an example
Mutant 1:
• main(argc,argv)• int argc, r, i;• char *argv[];• { r = 1;• for i = 1 to 3 do• if (atoi(argv[i-1]) > atoi(argv[r-1])) r = i;• printf(“Value of the rank is %d \n”, r);• exit(0); }
Test Case 1: input: 1 2 3 output: Value of the rank is ?Test Case 2: input: 1 2 1 output: Values of the rank is ?Test Case 3: input: 3 1 2 output: Value of the rank is ?
12
Mutation Testing: an example
Mutant 2:
• main(argc,argv)• int argc, r, i;• char *argv[];• { r = 1;• for i = 2 to 3 do• if (i-1) > atoi(argv[r-1])) r = i;• printf(“Value of the rank is %d \n”, r);• exit(0); }
Test Case 1: input: 1 2 3 output: Value of the rank is ?Test Case 2: input: 1 2 1 output: Values of the rank is ?Test Case 3: input: 3 1 2 output: Value of the rank is ?
13
Mutation Testing: an example
Mutant 3:
• main(argc,argv)• int argc, r, i;• char *argv[];• { r = 1;• for i = 2 to 3 do• if (atoi(argv[i-1]) >= atoi(argv[r-1])) r = i;• printf(“Value of the rank is %d \n”, r);• exit(0); }
Test Case 1: input: 1 2 3 output: Value of the rank is ?Test Case 2: input: 1 2 1 output: Values of the rank is ?Test Case 3: input: 3 1 2 output: Value of the rank is ?
14
Mutation Testing: an example
Mutation 4:
• main(argc,argv)• int argc, r, i;• char *argv[];• { r = 1;• for i = 2 to 3 do• if (atoi(argv[r-1]) > atoi(argv[r-1])) r = i;• printf(“Value of the rank is %d \n”, r);• exit(0); }
Test Case 1: input: 1 2 3 output: Value of the rank is ?Test Case 2: input: 1 2 1 output: Values of the rank is ?Test Case 3: input: 3 1 2 output: Value of the rank is ?
15
Mutation Testing: an exampleSummary:
• After Execution of modified programs against the test suite, you will get the results:
• Mutants 1 & 3: Programs will pass the test suite, i.e., mutants 1 & 3 are not killable
• Mutant 2: Program will fail test cases 2• Mutant 4: Program will fail test case 1 and test cases 2
• Mutation score is 50%, assuming mutants 1 & 3 non-equivalent
16
Mutation Testing: an exampleImprovements:
• The score is found to be low because we assumed mutants 1 & 3 are nonequivalent
• We need to show that mutants 1 and 3 are equivalent mutants or those are killable
• To show that those are killable, we need to add new test cases to kill these two mutants
17
Mutation Testing: an exampleImprovements:
• First, let us analyze mutant 1 in order to derive a “killer” test. The difference between P and mutant 1 is the starting point
• Mutant 1 starts with i = 1, whereas P starts with i = 2. There is no impact on the result r. Therefore, we conclude that mutant 1 is an equivalent mutant
• Second, if we add a fourth test case as follows:
Test Case 4:
input: 2 2 1
• Program P will produce the output “Value of the rank is 1” and mutant 3 will produce the output “Value of the rank is 2”
• Thus, this test data kills mutant 3, which give us a mutation score 100%
18
Mutation Testing: an example
Recap:
• Test Case 1:
input: 1 2 3
output: Value of the rank is 3• Test Case 2:
input: 1 2 1
output: Values of the rank is 2• Test Case 3:
input: 3 1 2
output: Value of the rank is 1• Test Case 4:
input: 2 2 1
output: Value of the rank is 1
which give us a mutation score 100% !!!
If process is not error-free, fix it
Test Mutants
The Mutation Process
Process
MutationMutationMutation Tests
Test Process
Create Mutants
Yes
Test CompleteNo
Any Live Mutants?
Problem with Tests?
Any Mutations that are caught by tests are killed
New Test Data
Fault Forecasting
20
Mutation testing
Fault ForecastingHow to determine number of remaining bugs?
Definition:Fault forecasting is conducted by performing an evaluation
of the system behavior with respect to fault occurrence or activation
Fault Injection technique:The idea is to inject (seed) some faults in the program and
calculate the remaining bugs based on detecting the seeded faults [Mills 1972]. Assuming that the probability of detecting the seeded and nonseeded faults are the same.
21
Fault Forecasting
22
Ns
Nd
Nr
nsnd
Fault Forecasting Example
23
Tools
24
Mutation testing
© Aditya P. Mathur 2009
Tools for mutation testing
• As with any other type of test adequacy assessment, mutation based assessment must be done with the help of a tool.
• There are few mutation testing tools available freely. Two such tools are Proteum for C from Professor Maldonado and muJava for Java from Professor Jeff Offutt. We are not aware of any commercially available tool for mutation testing. See the textbook (Aditya) for a more complete listing of mutation tools.
Q & A