I.T.A.K.E Unconference - Mutation testing to the rescue of your tests
GeeCON - Improve your tests with Mutation Testing
-
Upload
nicolas-frankel -
Category
Software
-
view
1.451 -
download
2
Transcript of GeeCON - Improve your tests with Mutation Testing
![Page 1: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/1.jpg)
Improve your tests with Mutation Testing
Nicolas Fränkel
![Page 2: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/2.jpg)
@nicolas_frankel 2
Me, Myself and I
Developer & ArchitectAs Consultant
Teacher/trainerBloggerSpeakerBook Author
![Page 3: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/3.jpg)
@nicolas_frankel 3
Shameless self-promotion
![Page 4: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/4.jpg)
@nicolas_frankel 4
My job
![Page 5: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/5.jpg)
Many kinds of testing
Unit TestingIntegration TestingEnd-to-end TestingPerformance TestingPenetration TestingExploratory Testingetc.
@nicolas_frankel 5
![Page 6: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/6.jpg)
Their only single goal
Ensure the Quality of the production code
@nicolas_frankel 6
![Page 7: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/7.jpg)
The problem
How to check the Quality of the testing code?
@nicolas_frankel 7
![Page 8: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/8.jpg)
Code coverage
“Code coverage is a measure used to describe the degree to which the source code of a program is tested”
--Wikipediahttp://en.wikipedia.org/wiki/
Code_coverage
@nicolas_frankel 8
![Page 9: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/9.jpg)
Measuring Code Coverage
Check whether a source code line is executed during a testOr Branch Coverage
@nicolas_frankel 9
![Page 10: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/10.jpg)
@nicolas_frankel 10
Computing Code Coverage
CC: Code Coverage(in percent)
Lexecuted: Number of executed lines of code
Ltotal: Number of total lines of code
![Page 11: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/11.jpg)
Java Tools for Code Coverage
JaCoCoCloverCoberturaetc.
@nicolas_frankel 11
![Page 12: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/12.jpg)
100% Code Coverage?
“Is 100% code coverage realistic? Of course it is. If you can write a line of code, you can write another that tests it.”
Robert Martin (Uncle Bob)https://twitter.com/unclebobmartin/status/
55966620509667328
@nicolas_frankel 12
![Page 13: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/13.jpg)
@nicolas_frankel 13
Assert-less testing
@Testpublic void add_should_add() { new Math().add(1, 1);}But, where is the assert?As long as the Code Coverage
is OK…
![Page 14: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/14.jpg)
Code coverage as a measure of test quality
Any metric can be gamed!Code coverage is a metric…
⇒ Code coverage can be gamedOn purposeOr by accident
@nicolas_frankel 14
![Page 15: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/15.jpg)
Code coverage as a measure of test quality
Code Coverage lulls you into a false sense of security…
@nicolas_frankel 15
![Page 16: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/16.jpg)
The problem still stands
Code coverage cannot ensure test qualityIs there another way?
Mutation Testing to the rescue!
@nicolas_frankel 16
![Page 17: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/17.jpg)
@nicolas_frankel 17
The Cast
William StrykerOriginal Source Code
Jason StrykerModified Source Code
a.k.a “The Mutant”
![Page 18: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/18.jpg)
@nicolas_frankel 18
public class Math { public int add(int i1, int i2) { return i1 + i2; }}
public class Math { public int add(int i1, int i2) { return i1 - i2; }}
![Page 19: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/19.jpg)
@nicolas_frankel 19
Standard testing
✔Execute Test
![Page 20: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/20.jpg)
@nicolas_frankel 20
Mutation testing
?Execute SAME Test
MUTATION
![Page 21: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/21.jpg)
@nicolas_frankel 21
Mutation testing
✗
✔Execute SAME Test
Execute SAME Test
Mutant Killed
Mutant Survived
![Page 22: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/22.jpg)
Killed or Surviving?
Surviving means changing the source code did not change the test resultIt’s bad!
Killed means changing the source code changed the test resultIt’s good
@nicolas_frankel 22
![Page 23: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/23.jpg)
@nicolas_frankel 23
Test the code
public class Math { public int add(int i1, int i2) { return i1 + i2; }}
@Testpublic void add_should_add() { new Math().add(1, 1);}
✔Execute Test
![Page 24: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/24.jpg)
@nicolas_frankel 24
Surviving mutant
public class Math { public int add(int i1, int i2) { return i1 - i2; }}
@Testpublic void add_should_add() { new Math().add(1, 1);}
✔Execute SAME Test
![Page 25: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/25.jpg)
@nicolas_frankel 25
Test the code
public class Math { public int add(int i1, int i2) { return i1 + i2; }}
@Testpublic void add_should_add() { int sum = new Math().add(1, 1); Assert.assertEquals(sum, 2);}
✔Execute Test
![Page 26: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/26.jpg)
@nicolas_frankel 26
Killed mutant
public class Math { public int add(int i1, int i2) { return i1 - i2; }}
@Testpublic void add_should_add() { int sum = new Math().add(1, 1); Assert.assertEquals(sum, 2);}
✗Execute SAME Test
![Page 27: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/27.jpg)
Mutation Testing in Java
PIT is a tool for Mutation testingAvailable as
Command-line toolAnt targetMaven plugin
@nicolas_frankel 27
![Page 28: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/28.jpg)
Mutators
Mutators are patterns applied to source code to produce mutations
@nicolas_frankel 28
![Page 29: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/29.jpg)
@nicolas_frankel 29
PIT mutators sample
Name Example source ResultConditionals Boundary > >=Negate Conditionals == !=Remove Conditionals foo == bar trueMath + -Increments foo++ foo--Invert Negatives -foo fooInline Constant static final FOO= 42 static final FOO =
43Return Values return true return falseVoid Method Call System.out.println("foo")Non Void Method Call long t =
System.currentTimeMillis()long t = 0
Constructor Call Date d = new Date() Date d = null;
![Page 30: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/30.jpg)
Important mutators
Conditionals BoundaryProbably a potential serious bug smell
if (foo > bar)
@nicolas_frankel 30
![Page 31: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/31.jpg)
Important mutators
Void Method Call
Assert.checkNotNull()connection.close()
@nicolas_frankel 31
![Page 32: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/32.jpg)
Remember
It’s not because the IDE generates code safely that it will never change
equals()hashCode()
@nicolas_frankel 32
![Page 33: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/33.jpg)
False positives
Mutation Testing is not 100% bulletproof
Might return false positivesBe cautious!
@nicolas_frankel 33
![Page 34: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/34.jpg)
@nicolas_frankel 34
Enough talk…
Time for DEMO
![Page 35: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/35.jpg)
Drawbacks
SlowSluggishCrawlingSulkyLethargicetc.
@nicolas_frankel 39
![Page 36: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/36.jpg)
Metrics (kind of)
On joda-moneymvn clean test-compilemvn surefire:test
Total time: 2.181 s
mvn pit-test...Total time: 48.634 s
@nicolas_frankel 40
![Page 37: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/37.jpg)
Why so slow?
Analyze test codeFor each class under test
For each mutatorCreate mutation
For each mutationRun testAnalyze resultAggregate results
@nicolas_frankel 41
![Page 38: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/38.jpg)
Workarounds
This is not acceptable in a normal test run
But there are workarounds
@nicolas_frankel 42
![Page 39: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/39.jpg)
@nicolas_frankel 43
Set mutators
<configuration> <mutators> <mutator> CONSTRUCTOR_CALLS </mutator> <mutator> NON_VOID_METHOD_CALLS </mutator> </mutators></configuration>
![Page 40: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/40.jpg)
@nicolas_frankel 44
Set target classes
<configuration> <targetClasses> <param>ch.frankel.pit*</param> </targetClasses></configuration>
![Page 41: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/41.jpg)
@nicolas_frankel 45
Set target tests
<configuration> <targetTests> <param>ch.frankel.pit*</param> </targetTests></configuration>
![Page 42: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/42.jpg)
@nicolas_frankel 46
Dependency distance
1 2
![Page 43: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/43.jpg)
@nicolas_frankel 47
Limit dependency distance
<configuration> <maxDependencyDistance> 4 </maxDependencyDistance></configuration>
![Page 44: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/44.jpg)
@nicolas_frankel 48
Limit number of mutations
<configuration> <maxMutationsPerClass> 10 </maxMutationsPerClass></configuration>
![Page 45: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/45.jpg)
Use MutationFilter
@nicolas_frankel 49
![Page 46: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/46.jpg)
PIT extension points
Must be packaged in JARHave Implementation-Vendor and
Implementation-Title in MANIFEST.MF that match PIT’s
Set on the classpathUse Java’s Service Provider feature
@nicolas_frankel 50
![Page 47: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/47.jpg)
Service Provider
Inversion of controlSince Java 1.3!
Text file located in META-INF/services
InterfaceName of the file
Implementation classContent of the file
@nicolas_frankel 51
![Page 48: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/48.jpg)
@nicolas_frankel 52
Sample structure
JARch.frankel.lts.Sample
![Page 49: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/49.jpg)
@nicolas_frankel 53
Service Provider API
![Page 50: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/50.jpg)
@nicolas_frankel 54
Service Provider sample
ServiceLoader<ISample> loaders = ServiceLoader.load(ISample.class);
for (ISample sample: loaders) {// Use the sample
}
![Page 51: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/51.jpg)
Output formats
Out-of-the-boxHTMLXMLCSV
@nicolas_frankel 55
![Page 52: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/52.jpg)
@nicolas_frankel 56
Output formats
<configuration> <outputFormats> <outputFormat>XML</outputFormat> <outputFormat>HTML</outputFormat> </outputFormats></configuration>
![Page 53: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/53.jpg)
@nicolas_frankel 57
![Page 54: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/54.jpg)
Mutation Filter
Remove mutations from the list of available mutations for a class
@nicolas_frankel 58
![Page 55: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/55.jpg)
@nicolas_frankel 59
![Page 56: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/56.jpg)
@nicolas_frankel 60
Time for DEMO
![Page 57: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/57.jpg)
@nicolas_frankel 61
Don’t bind to test phase!
<plugin> <groupId>org.pitest</groupId> <artifactId>pitest-maven</artifactId> <executions> <execution> <goals> <goal>mutationCoverage</goal> </goals> <phase>test</phase> </execution> </executions></plugin>
![Page 58: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/58.jpg)
@nicolas_frankel 62
Use scmMutationCoverage
mvn \org.pitest:pitest-maven:scmMutationCoverage \
-DtimestampedReports=false
maven-scm-
plugin
must be configured!
![Page 59: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/59.jpg)
@nicolas_frankel 63
Do use on Continuous Integration servers
mvn \org.pitest:pitest-maven:mutationCoverage \-DtimestampedReports=false
![Page 60: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/60.jpg)
Is Mutation Testing the Silver Bullet?
Sorry, no!It only
Checks the relevance of your unit testsPoints out potential bugs
@nicolas_frankel 64
![Page 61: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/61.jpg)
What it doesn’t do
Validate the assembled applicationIntegration Testing
Check the performancePerformance Testing
Look out for display bugsEnd-to-end testing
Etc.
@nicolas_frankel 65
![Page 62: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/62.jpg)
Testing is about ROI
Don’t test to achieve 100% coverageTest because it saves money in the
long runPrioritize:
Business-critical codeComplex code
@nicolas_frankel 66
![Page 63: GeeCON - Improve your tests with Mutation Testing](https://reader035.fdocuments.net/reader035/viewer/2022062503/58e791fa1a28abc52e8b5199/html5/thumbnails/63.jpg)
@nicolas_frankel 67
Q&A
@nicolas_frankelhttp://blog.frankel.ch/https://leanpub.com/
integrationtest/