Better code through making bugs
-
Upload
seb-rose -
Category
Technology
-
view
804 -
download
3
description
Transcript of Better code through making bugs
Tuesday, 22 April 14
Be#er code through making bugs
Seb Rose
Claysnow Limited
@sebrose
Tuesday, 22 April 14
Massive thanks to:
3
Henry Colespitest
Filip van Laenenmutant
Tuesday, 22 April 14
How do you assure the quality of your test suite?
Tuesday, 22 April 14
Only employ coding gods
How do you assure the quality of your test suite?
Tuesday, 22 April 14
Only employ coding gods
TDD will protect us
How do you assure the quality of your test suite?
Tuesday, 22 April 14
Only employ coding gods
TDD will protect us
Peer review
How do you assure the quality of your test suite?
Tuesday, 22 April 14
Only employ coding gods
TDD will protect us
Peer review
QA will catch anything that we miss
How do you assure the quality of your test suite?
Tuesday, 22 April 14
Only employ coding gods
TDD will protect us
Peer review
QA will catch anything that we miss
Good code coverage
How do you assure the quality of your test suite?
Tuesday, 22 April 14
How do you assure the quality of your test suite?
Tuesday, 22 April 14
Code coverage?
How do you assure the quality of your test suite?
Tuesday, 22 April 14
Code coverage?
Line coverage?
How do you assure the quality of your test suite?
Tuesday, 22 April 14
Code coverage?
Line coverage?Branch coverage?
How do you assure the quality of your test suite?
Tuesday, 22 April 14
Code coverage?
Line coverage?Branch coverage?
Statement coverage?
How do you assure the quality of your test suite?
Tuesday, 22 April 14
Does good coverage guarantee that:
Tuesday, 22 April 14
• I can safely refactor my tests?
Does good coverage guarantee that:
Tuesday, 22 April 14
• I can safely refactor my tests?
• I can trust a test suite I inherited?
Does good coverage guarantee that:
Tuesday, 22 April 14
• I can safely refactor my tests?
• I can trust a test suite I inherited?
• My team are writing effective tests?
Does good coverage guarantee that:
Tuesday, 22 April 14
• I can safely refactor my tests?
• I can trust a test suite I inherited?
• My team are writing effective tests?
• I've retrofitted enough tests to protect my legacy code?
Does good coverage guarantee that:
Tuesday, 22 April 14
• I can safely refactor my tests?
• I can trust a test suite I inherited?
• My team are writing effective tests?
• I've retrofitted enough tests to protect my legacy code?
Does good coverage guarantee that:
NO, IT
DOESN’T
Tuesday, 22 April 14
Coverage tells us nothing about the quality of our tests
Tuesday, 22 April 14
Coverage tells us nothing about the quality of our tests
It tells us which statements have been run by our tests
Tuesday, 22 April 14
Coverage tells us nothing about the quality of our tests
It tells us which statements have NOT been tested
Tuesday, 22 April 14
Coverage tells us nothing about the quality of our tests
... but it is very useful for something else
Tuesday, 22 April 14
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
Tuesday, 22 April 14
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
Tuesday, 22 April 14
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
Tuesday, 22 April 14
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
Tuesday, 22 April 14
@Test public void shouldStartWithEmptyCount() { assertEquals(0,testee.currentCount()); }
@Test public void shouldCountIntegersAboveTen() { testee.count(11); assertEquals(1,testee.currentCount()); } @Test public void shouldNotCountIntegersBelowTen() { testee.count(9); assertEquals(0,testee.currentCount()); }
Tuesday, 22 April 14
Lipton, “Fault diagnosis in computer programs”,1971
“If we want to know if a test suite has properly checked some code
deliberately introduce a bug”
Tuesday, 22 April 14
The deliberate introduction of a bug by changing the code under test
Mutation:
Tuesday, 22 April 14
A version of the code under test that has had a single mutation
Mutant:
Tuesday, 22 April 14
Tuesday, 22 April 14
Mutation test:
Tuesday, 22 April 14
Run your test suite
Mutation test:
Tuesday, 22 April 14
Run your test suite
If any test fails, the mutant has been killed
Mutation test:
Tuesday, 22 April 14
Run your test suite
If any test fails, the mutant has been killed
If no test fails, the mutant has survived
Mutation test:
Tuesday, 22 April 14
Mutation testing:
Tuesday, 22 April 14
Generate lots of mutants
Mutation testing:
Tuesday, 22 April 14
Generate lots of mutants
Run each one through your test suite
Mutation testing:
Tuesday, 22 April 14
Generate lots of mutants
Run each one through your test suite
Record and interpret the results
Mutation testing:
Tuesday, 22 April 14
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
Tuesday, 22 April 14
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
if ( i > 10 ) {
Tuesday, 22 April 14
Mutation operators:
• Conditionals Boundary Mutator• Negate Conditionals Mutator• Remove Conditionals Mutator• Math Mutator• Increments Mutator• Invert Negatives Mutator• Inline Constant Mutator• Return Values Mutator• Void Method Calls Mutator• Non Void Method Calls Mutator• Constructor Calls Mutator
Tuesday, 22 April 14
“Generated mutants are similar to real faults”
Andrews, Briand, Labiche, ICSE 2005
Tuesday, 22 April 14
“In practice, if the software contains a fault, there will usually be a set of mutants that can only be killed by a test case that also detects that fault.”
Geist et. al., “Estimation and Enhancement of Real-time Software Reliability through Mutation Analysis,” 1992
Tuesday, 22 April 14
“Complex faults are coupled to simple faults in such a way that a test data set that detects all simple faults in a program will detect most complex faults.”
K. Wah, “Fault Coupling in Finite Bijective Functions,” 1995
Tuesday, 22 April 14
Poor performance
Equivalent mutations
Why isn’t mutation testing widely used?
Tuesday, 22 April 14
How bad is performance?
Tuesday, 22 April 14
How bad is performance?
Joda Time, consider, let us
Tuesday, 22 April 14
Joda Time is a ...
Tuesday, 22 April 14
small library for dealing with dates and times
Joda Time is a ...
Tuesday, 22 April 14
small library for dealing with dates and times
68k lines of code
Joda Time is a ...
Tuesday, 22 April 14
small library for dealing with dates and times
68k lines of code70k lines of test code
Joda Time is a ...
Tuesday, 22 April 14
small library for dealing with dates and times
68k lines of code70k lines of test code
Takes about 10 seconds to compile
Joda Time is a ...
Tuesday, 22 April 14
small library for dealing with dates and times
68k lines of code70k lines of test code
Takes about 10 seconds to compileTakes about 16 seconds to run the unit tests
Joda Time is a ...
Tuesday, 22 April 14
Tuesday, 22 April 14
Let’s use 10 mutation operators
Tuesday, 22 April 14
Let’s use 10 mutation operatorsassume about 10k mutations
Tuesday, 22 April 14
Let’s use 10 mutation operatorsassume about 10k mutations
If it takes 1 second to compile each one
Tuesday, 22 April 14
Let’s use 10 mutation operatorsassume about 10k mutations
If it takes 1 second to compile each one 2.5 hours to generate the mutants
Tuesday, 22 April 14
Let’s use 10 mutation operatorsassume about 10k mutations
If it takes 1 second to compile each one 2.5 hours to generate the mutants
Run the test suite for each mutant
Tuesday, 22 April 14
Let’s use 10 mutation operatorsassume about 10k mutations
If it takes 1 second to compile each one 2.5 hours to generate the mutants
Run the test suite for each mutant10k x 16 seconds = 44.5 hours
Tuesday, 22 April 14
pitest manipulates the byte code directly
10k mutants generated < 1 second
Don’t compile!
Tuesday, 22 April 14
Run fewer tests!
Tuesday, 22 April 14
Stop when a test fails
Run fewer tests!
Tuesday, 22 April 14
Stop when a test failscan easily halve the run time
Run fewer tests!
Tuesday, 22 April 14
Stop when a test failscan easily halve the run time
Choose your tests carefully
Run fewer tests!
Tuesday, 22 April 14
Stop when a test failscan easily halve the run time
Choose your tests carefullynot every test can kill every mutant
Run fewer tests!
Tuesday, 22 April 14
Stop when a test failscan easily halve the run time
Choose your tests carefullynot every test can kill every mutant
Parallelise the test runner
Run fewer tests!
Tuesday, 22 April 14
Stop when a test failscan easily halve the run time
Choose your tests carefullynot every test can kill every mutant
Parallelise the test runneryour tests are unit tests, right?
Run fewer tests!
Tuesday, 22 April 14
Choosing your tests well is critical
Tuesday, 22 April 14
Each mutant can only ever be killed
Choosing your tests well is critical
Tuesday, 22 April 14
Each mutant can only ever be killed
by a subset of the tests
Choosing your tests well is critical
Tuesday, 22 April 14
Each mutant can only ever be killed
by a subset of the tests
Every other test run is waste
Choosing your tests well is critical
Tuesday, 22 April 14
How can we know which tests might kill
any given mutant?
Tuesday, 22 April 14
How can we know which tests might kill
any given mutant?
Naming conventions?
Tuesday, 22 April 14
How can we know which tests might kill
any given mutant?
Naming conventions?Static analysis?
Tuesday, 22 April 14
How can we know which tests might kill
any given mutant?
Naming conventions?Static analysis?
COVERAGE DATA!
Tuesday, 22 April 14
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
Tuesday, 22 April 14
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
Tuesday, 22 April 14
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
Tuesday, 22 April 14
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
Tuesday, 22 April 14
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
shouldCountIntegersAboveTenshouldNotCountIntegersBelowTenshouldCountIntegersOfExactlyTen
Tuesday, 22 April 14
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
shouldCountIntegersAboveTenshouldNotCountIntegersBelowTenshouldCountIntegersOfExactlyTen
shouldCountIntegersAboveTenshouldCountIntegersOfExactlyTen
Tuesday, 22 April 14
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
shouldCountIntegersAboveTenshouldNotCountIntegersBelowTenshouldCountIntegersOfExactlyTen
shouldCountIntegersAboveTenshouldCountIntegersOfExactlyTen
not covered by any test
Tuesday, 22 April 14
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
shouldCountIntegersAboveTenshouldNotCountIntegersBelowTenshouldCountIntegersOfExactlyTen
shouldCountIntegersAboveTenshouldCountIntegersOfExactlyTen
not covered by any test
shouldStartWithEmptyCount does not cover any of the lines shown
Tuesday, 22 April 14
Joda Time on my machine, with 2 threads:
- Timings> scan classpath : < 1 second> coverage and dependency analysis : 59 seconds> build mutation tests : 1 seconds> run mutation analysis : 8 minutes and 21 seconds> Total : 9 minutes and 22 seconds
- Statistics>> Generated 9922 mutations Killed 7833 (79%)>> Ran 117579 tests (11.85 tests per mutation)
Tuesday, 22 April 14
Why only 79% killed?
Tuesday, 22 April 14
Missing test cases
Why only 79% killed?
Tuesday, 22 April 14
Missing test cases
Time outs
Why only 79% killed?
Tuesday, 22 April 14
Missing test cases
Time outs
Equivalent mutations
Why only 79% killed?
Tuesday, 22 April 14
public void someLogic(int i) { if (i <= 100) { throw new IllegalArgumentException(); }
if (i >= 100) { doSomething(); }}
Equivalent mutant:
Tuesday, 22 April 14
public void someLogic(int i) { if (i <= 100) { throw new IllegalArgumentException(); }
if (i >= 100) { doSomething(); }}
if (i > 100) {
Equivalent mutant:
Tuesday, 22 April 14
public void someLogic(int i) { if (i <= 100) { throw new IllegalArgumentException(); }
if (i >= 100) { doSomething(); }}
if (i > 100) {
i can never be 100 here
Equivalent mutant:
Tuesday, 22 April 14
Maybe we should have written:
public void someLogic(int i) { if (i <= 100) { throw new IllegalArgumentException(); }
doSomething();}
Tuesday, 22 April 14
Some causes of equivalence are:
Tuesday, 22 April 14
Dead/useless code
Some causes of equivalence are:
Tuesday, 22 April 14
Dead/useless codeNon-functional modifications
Some causes of equivalence are:
Tuesday, 22 April 14
Dead/useless codeNon-functional modificationsUnsatisfiable guards
Some causes of equivalence are:
Tuesday, 22 April 14
Dead/useless codeNon-functional modificationsUnsatisfiable guardsInternal state
Some causes of equivalence are:
Tuesday, 22 April 14
Dead/useless codeNon-functional modificationsUnsatisfiable guardsInternal state
Some causes of equivalence are:
This can help us improve our code
Tuesday, 22 April 14
Mutants that survive need to bechecked manually
to determine if they are equivalent
Tuesday, 22 April 14
Is this a show-stopper?
Tuesday, 22 April 14
Is this a show-stopper?
Not all that common in practice
Tuesday, 22 April 14
Is this a show-stopper?
Not all that common in practice
TDD helps prevent equivalents
Tuesday, 22 April 14
Is this a show-stopper?
Not all that common in practice
TDD helps prevent equivalents
Tolerate a few survivors
Tuesday, 22 April 14
What about really large code bases?
Tuesday, 22 April 14
What about really large code bases?
Mutation testing can take a long time:
Tuesday, 22 April 14
What about really large code bases?
Mutation testing can take a long time:Use fewer mutants
Tuesday, 22 April 14
What about really large code bases?
Mutation testing can take a long time:Use fewer mutantsFilter the candidates
Tuesday, 22 April 14
What about really large code bases?
Mutation testing can take a long time:Use fewer mutantsFilter the candidatesRun infrequently
Tuesday, 22 April 14
Get the developers to mutation test
Tuesday, 22 April 14
Get the developers to mutation test
Only on code they are working on
Tuesday, 22 April 14
Get the developers to mutation test
Only on code they are working on
Faster feedback
Tuesday, 22 April 14
Get the developers to mutation test
Only on code they are working on
Faster feedback
Improves design of the code
Tuesday, 22 April 14
Get your CI server to run the mutation test
Tuesday, 22 April 14
Get your CI server to run the mutation test
Over whole codebase if quick enough
Tuesday, 22 April 14
Get your CI server to run the mutation test
Over whole codebase if quick enough
Use SCM integration to identify subset
Tuesday, 22 April 14
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
Use mutation testing from day 1
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
Use mutation testing from day 1 Start on a small code base
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
Use mutation testing from day 1 Start on a small code baseKeep number of unit tests per class low
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
Use mutation testing from day 1 Start on a small code baseKeep number of unit tests per class lowHave small classes
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
Use mutation testing from day 1 Start on a small code baseKeep number of unit tests per class lowHave small classesSelect a good tool:
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
Use mutation testing from day 1 Start on a small code baseKeep number of unit tests per class lowHave small classesSelect a good tool:
Configurable
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
Use mutation testing from day 1 Start on a small code baseKeep number of unit tests per class lowHave small classesSelect a good tool:
ConfigurableFlexible
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
Use mutation testing from day 1 Start on a small code baseKeep number of unit tests per class lowHave small classesSelect a good tool:
ConfigurableFlexibleIdentifies surviving the mutant
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
Believe the tool
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
Believe the toolFix the problem
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
Believe the toolFix the problemDon't turn mutation testing off
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
Believe the toolFix the problemDon't turn mutation testing offLess code
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
Believe the toolFix the problemDon't turn mutation testing offLess codeMore unit tests
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
Believe the toolFix the problemDon't turn mutation testing offLess codeMore unit tests
More intelligent unit tests
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
Available tools
http://en.wikipedia.org/wiki/Mutation_testing#External_links
Tuesday, 22 April 14
Ruby: Heckle, Mutant
Available tools
http://en.wikipedia.org/wiki/Mutation_testing#External_links
Tuesday, 22 April 14
Ruby: Heckle, MutantJava: pitest, Jumble, Jester
Available tools
http://en.wikipedia.org/wiki/Mutation_testing#External_links
Tuesday, 22 April 14
Ruby: Heckle, MutantJava: pitest, Jumble, JesterC#: Nester, NinjaTurtle, Cream
Available tools
http://en.wikipedia.org/wiki/Mutation_testing#External_links
Tuesday, 22 April 14
Ruby: Heckle, MutantJava: pitest, Jumble, JesterC#: Nester, NinjaTurtle, CreamPython: Pester
Available tools
http://en.wikipedia.org/wiki/Mutation_testing#External_links
Tuesday, 22 April 14
Open sourceWorks with Java 5, 6, 7Works with all mocking frameworks including PowerMockIntegrates with Maven, Ant, Gradle & SBTPlugins for Eclipse & IntelliJPlugins for Jenkins & SonarQubeReleases every 3 months or so since 2011
Tuesday, 22 April 14
• The Ladders - New York• Sky - Livingston• Insurance companies• Investment banks• Biotech companies• Norway's e-voting system
Tuesday, 22 April 14
• The Ladders - New York• Sky - Livingston• Insurance companies• Investment banks• Biotech companies• Norway's e-voting system
Tuesday, 22 April 14
• The Ladders - New York• Sky - Livingston• Insurance companies• Investment banks• Biotech companies• Norway's e-voting system
Tuesday, 22 April 14
-JVM
Seb Rose,
Tuesday, 22 April 14
-JVM
Seb Rose,
Availa
ble 20
14
(hopef
ully)
Tuesday, 22 April 14