Mutation Testing as a Safety Net for Test Code Refactoring · Mutation Testing as a Safety Net for...
Transcript of Mutation Testing as a Safety Net for Test Code Refactoring · Mutation Testing as a Safety Net for...
Mutation Testing as a Safety Net for
Test Code Refactoring
Ali ParsaiAlessandro MurgiaQuinten D. SoetensSerge Demeyer
2
“Refactoring is changing a system to improve its internal structure without altering its external behavior.” – Fowler et al.
3
Does refactoring preserve the behavior?
Perform refactoring
Run the test suite
Do all tests pass?
Production Code
4
Does refactoring preserve the behavior?
Perform refactoring
Test Code
5
SolutionA metric to verify the preservation of the test
suite behavior pre- and post- refactoring
6
Mutation Coverage
Branch Coverage
Options
7
Branch Coverage
33%
8
Mutation CoverageProcedure
Example
75%
9
Branch Coverage+ Can be calculated fast+ Very good tool support – Easily integrable – Not accurate for defining behavior
10
Mutation Coverage+ More accurate + Adaptable by customizing the fault model – Computationally intensive – Poor tool support, hard to integrate
Works on Java source code
Works regardless of build system and testing framework
11
12
Main
JavaRead JavaParse JavaMutate ReportGenerator
Antlr-generatedJava Parser
CustomizedAntlr4 Runtime
13
LittleDarwin Project Report
14
LittleDarwin File Report
15
Mutation testing tool → LittleDarwinBranch coverage tool → JaCoCoToy example → SalaryReal project → AddThis Codec
Experiments
16
Salary class diagram
Toy Example
17
Toy Example
18
Toy Example
Pre- Refactoring
19
Toy Example
Post- Refactoring
20
Branch coverage and percentage of passed tests remain the sameMutation coverage changes
Toy Example
21
The mutant that makes a difference can be traced back to the problematic test
Toy Example
Pre- Version
22
Real Project
Found a version with refactorings in the test code
Identified the refactorings using RefFinder
Backported the changes in the test code to the previous version
ProductionCode
TestCode
Post- Version
ProductionCode
BackportedTest Code
23
Real Project
24
Mutation Coverage remains the same for all classes
Real Project
Limitations
25
One can catch smaller fishes with a tighter net
ConclusionMutation testing can act as a safety net for test suite refactoring by detecting behavior changes
26
Mutation testing cannot guaranty the preservation of the test suite behavior in every situationThis method can be used to verify automated refactoring tools
27
Does refactoring preserve the behavior?
Perform refactoring
Test Code
pre- and post- refactoring
Does the mutation coverage stay the
same?