Kill the Mutants - a better way to test your tests by Roy van Rijn
-
Upload
baruch-sadogursky -
Category
Software
-
view
68 -
download
1
description
Transcript of Kill the Mutants - a better way to test your tests by Roy van Rijn
![Page 1: Kill the Mutants - a better way to test your tests by Roy van Rijn](https://reader038.fdocuments.net/reader038/viewer/2022110310/559444891a28ab01308b47d6/html5/thumbnails/1.jpg)
KILL THE MUTANTSa better way to test your tests
![Page 2: Kill the Mutants - a better way to test your tests by Roy van Rijn](https://reader038.fdocuments.net/reader038/viewer/2022110310/559444891a28ab01308b47d6/html5/thumbnails/2.jpg)
SHOW OF HANDSlet's do a
• Unit testing
• Continuous integration
• Code coverage
• Mutation testing
![Page 3: Kill the Mutants - a better way to test your tests by Roy van Rijn](https://reader038.fdocuments.net/reader038/viewer/2022110310/559444891a28ab01308b47d6/html5/thumbnails/3.jpg)
CODE COVERAGE• Measure the lines (or branches) that are executed during testing
![Page 4: Kill the Mutants - a better way to test your tests by Roy van Rijn](https://reader038.fdocuments.net/reader038/viewer/2022110310/559444891a28ab01308b47d6/html5/thumbnails/4.jpg)
CODE COVERAGE
![Page 5: Kill the Mutants - a better way to test your tests by Roy van Rijn](https://reader038.fdocuments.net/reader038/viewer/2022110310/559444891a28ab01308b47d6/html5/thumbnails/5.jpg)
QUIS CUSTODIET IPSOS CUSTODES?Who watches the watchmen?
![Page 6: Kill the Mutants - a better way to test your tests by Roy van Rijn](https://reader038.fdocuments.net/reader038/viewer/2022110310/559444891a28ab01308b47d6/html5/thumbnails/6.jpg)
MUTATION TESTING
• Proposed by Richard J. Lipton in 1971
• Surge of interest in the 1980s
• Time to revive this interest!
![Page 7: Kill the Mutants - a better way to test your tests by Roy van Rijn](https://reader038.fdocuments.net/reader038/viewer/2022110310/559444891a28ab01308b47d6/html5/thumbnails/7.jpg)
TERMINOLOGY: MUTATION
![Page 8: Kill the Mutants - a better way to test your tests by Roy van Rijn](https://reader038.fdocuments.net/reader038/viewer/2022110310/559444891a28ab01308b47d6/html5/thumbnails/8.jpg)
TERMINOLOGY: MUTANT
• A mutant is a mutated version of your class
![Page 9: Kill the Mutants - a better way to test your tests by Roy van Rijn](https://reader038.fdocuments.net/reader038/viewer/2022110310/559444891a28ab01308b47d6/html5/thumbnails/9.jpg)
MUTATION TESTING• Generate (a lot of) mutants of your codebase
• Run (some of) your unit tests
• Check the outcome!
![Page 10: Kill the Mutants - a better way to test your tests by Roy van Rijn](https://reader038.fdocuments.net/reader038/viewer/2022110310/559444891a28ab01308b47d6/html5/thumbnails/10.jpg)
OUTCOME #1: KILLED
• A mutant is killed if a test fails (detecting the mutated code)
• This proves the mutated code is properly tested
![Page 11: Kill the Mutants - a better way to test your tests by Roy van Rijn](https://reader038.fdocuments.net/reader038/viewer/2022110310/559444891a28ab01308b47d6/html5/thumbnails/11.jpg)
OUTCOME #2: LIVED
• A mutant didn’t trigger a failing test…
![Page 12: Kill the Mutants - a better way to test your tests by Roy van Rijn](https://reader038.fdocuments.net/reader038/viewer/2022110310/559444891a28ab01308b47d6/html5/thumbnails/12.jpg)
OTHER OUTCOMES
• TIMED OUT: Infinite loop
• NON-VIABLE: Illegal bytecode
• MEMORY ERROR: Out of memory
• RUN ERROR: Other…
![Page 13: Kill the Mutants - a better way to test your tests by Roy van Rijn](https://reader038.fdocuments.net/reader038/viewer/2022110310/559444891a28ab01308b47d6/html5/thumbnails/13.jpg)
TOOLING• µJava: http://cs.gmu.edu/~offutt/mujava/ (inactive)
• Jester : http://jester.sourceforge.net/ (inactive)
• Jumble: http://jumble.sourceforge.net/ (inactive)
• javaLanche: http://www.st.cs.uni-saarland.de/mutation/ (inactive)
• PIT: http://pitest.org/
![Page 14: Kill the Mutants - a better way to test your tests by Roy van Rijn](https://reader038.fdocuments.net/reader038/viewer/2022110310/559444891a28ab01308b47d6/html5/thumbnails/14.jpg)
USING PIT
• Manipulates bytecode with ASM (at runtime)
• Mutants are never stored
![Page 15: Kill the Mutants - a better way to test your tests by Roy van Rijn](https://reader038.fdocuments.net/reader038/viewer/2022110310/559444891a28ab01308b47d6/html5/thumbnails/15.jpg)
MUTATORS
• Example of mutators:
> into >=< into <=>= into ><= into <
![Page 16: Kill the Mutants - a better way to test your tests by Roy van Rijn](https://reader038.fdocuments.net/reader038/viewer/2022110310/559444891a28ab01308b47d6/html5/thumbnails/16.jpg)
MUTATION TESTING IS SLOW?
![Page 17: Kill the Mutants - a better way to test your tests by Roy van Rijn](https://reader038.fdocuments.net/reader038/viewer/2022110310/559444891a28ab01308b47d6/html5/thumbnails/17.jpg)
WHICH TESTS TO RUN?
• PIT uses code coverage!
![Page 18: Kill the Mutants - a better way to test your tests by Roy van Rijn](https://reader038.fdocuments.net/reader038/viewer/2022110310/559444891a28ab01308b47d6/html5/thumbnails/18.jpg)
WHICH TESTS TO RUN?
• A mutation is on a line covered by 3 tests? Only run those.
![Page 19: Kill the Mutants - a better way to test your tests by Roy van Rijn](https://reader038.fdocuments.net/reader038/viewer/2022110310/559444891a28ab01308b47d6/html5/thumbnails/19.jpg)
EASY TO USE:
• Standalone Java process
• Build: Ant and Maven plugin
• CI: Sonarqube and Gradle plugin
• IDE: Eclipse and IntelliJ plugin
![Page 20: Kill the Mutants - a better way to test your tests by Roy van Rijn](https://reader038.fdocuments.net/reader038/viewer/2022110310/559444891a28ab01308b47d6/html5/thumbnails/20.jpg)
SUMMARY
• Code coverage: BAD
• Mutation testing: GOOD!
• Automatic, easy to add, and meaningful!