Can You Trust Your Tests? (Agile Tour 2015 Kaunas)
-
Upload
vaidas-pilkauskas -
Category
Software
-
view
427 -
download
1
Transcript of Can You Trust Your Tests? (Agile Tour 2015 Kaunas)
![Page 1: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/1.jpg)
CanYou Trust Your Tests?
2015 Vaidas Pilkauskas & Tadas Ščerbinskas
![Page 2: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/2.jpg)
Van Halen
![Page 3: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/3.jpg)
Band Tour Rider
![Page 4: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/4.jpg)
Van Halen’s 1982 Tour Rider
![Page 5: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/5.jpg)
![Page 6: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/6.jpg)
Agenda
1. Test quality & code coverage2. Mutation testing in theory3. Mutation testing in practice
![Page 7: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/7.jpg)
Prod vs. Test code quality
Code has bugs.
Tests are code.
Tests have bugs.
![Page 8: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/8.jpg)
![Page 9: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/9.jpg)
Test quality
Readable
Focused
Concise
Well named
![Page 10: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/10.jpg)
“Program testing can be used to show the presence of bugs, but never to
show their absence!”- Edsger W. Dijkstra
![Page 11: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/11.jpg)
Code Coverage
![Page 12: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/12.jpg)
Types of Code Coverage
Lines
Branches
Instructions
Cyclomatic Complexity
Methods
& more
![Page 13: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/13.jpg)
![Page 14: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/14.jpg)
Lines
string foo() { return "a" + "b"}
assertThat(a.foo(), is("ab"))
![Page 15: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/15.jpg)
Lines
string foo(boolean arg) { return arg ? "a" : "b"}
assertThat(a.foo(true), is("a"))
![Page 16: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/16.jpg)
Branches
string foo(boolean arg) { return arg ? "a" : "b"}
assertThat(a.foo(true), is("a"))
assertThat(a.foo(false), is("b"))
![Page 17: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/17.jpg)
SUCCESS: 26/26 (100%) Tests passed
![Page 18: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/18.jpg)
Can you trust 100% coverage?
Code coverage can only show what is not tested.
For interpreted languages 100% code coverage is kind of like full compilation.
![Page 19: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/19.jpg)
Code Coverage can be gamed
On purpose or by accident
![Page 20: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/20.jpg)
Mutation testing
![Page 21: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/21.jpg)
Mutation testing
Changes your program code and expects your tests to fail.
![Page 22: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/22.jpg)
What exactly is a mutation?
def isFoo(a) { return a == foo}
def isFoo(a) { return a != foo}
def isFoo(a) { return true}
def isFoo(a) { return null}
>>>
![Page 23: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/23.jpg)
Terminology
Applying a mutation to some code creates a mutant.
If test passes - mutant has survived.
If test fails - mutant is killed.
![Page 24: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/24.jpg)
Failing is the new passing
![Page 25: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/25.jpg)
array = [a, b, c]
max(array) == ???
![Page 26: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/26.jpg)
// testmax([0]) == 0 ✘
![Page 27: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/27.jpg)
// testmax([0]) == 0 ✔
// implementationmax(a) { return 0}
![Page 28: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/28.jpg)
// testmax([0]) == 0 ✔max([1]) == 1 ✘
// implementationmax(a) { return 0}
![Page 29: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/29.jpg)
// testmax([0]) == 0 ✔max([1]) == 1 ✔
// implementationmax(a) { return a.first}
![Page 30: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/30.jpg)
// testmax([0]) == 0 ✔max([1]) == 1 ✔max([0, 2]) == 2 ✘
// implementationmax(a) { return a.first}
![Page 31: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/31.jpg)
// testmax([0]) == 0 ✔max([1]) == 1 ✔max([0, 2]) == 2 ✔
// implementationmax(a) { m = a.first for (e in a) if (e > m) m = e return m}
![Page 32: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/32.jpg)
Coverage
![Page 33: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/33.jpg)
Mutation// testmax([0]) == 0 ✔max([1]) == 1 ✔max([0, 2]) == 2 ✔
// implementationmax(a) { m = a.first for (e in a) if (e > m) m = e return m}
![Page 34: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/34.jpg)
Mutation// testmax([0]) == 0 ✔max([1]) == 1 ✔max([0, 2]) == 2 ✔
// implementationmax(a) { m = a.first for (e in a) if (true) m = e return m}
![Page 35: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/35.jpg)
// testmax([0]) == 0 ✔max([1]) == 1 ✔max([0, 2]) == 2 ✔
// implementationmax(a) { return a.last}
![Page 36: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/36.jpg)
// testmax([0]) == 0 ✔max([1]) == 1 ✔max([0, 2]) == 2 ✔max([2, 1]) == 2 ✘
// implementationmax(a) { return a.last}
![Page 37: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/37.jpg)
// implementationmax(a) { m = a.first for (e in a) if (e > m) m = e return m}
// testmax([0]) == 0 ✔max([1]) == 1 ✔max([0, 2]) == 2 ✔max([2, 1]) == 2 ✔
![Page 38: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/38.jpg)
Baby steps matter
![Page 39: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/39.jpg)
Tests’ effectiveness is measured by number of killed mutants by
your test suite.
![Page 40: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/40.jpg)
It’s like hiring a white-hat hacker to try to break into your server and making sure you detect it.
![Page 41: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/41.jpg)
What if mutant survives
● Simplify your code● Add additional tests
● TDD - minimal amount of code to pass the test
![Page 42: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/42.jpg)
Challenges
1. High computation cost - slow2. Equivalent mutants - false negatives3. Infinite loops
![Page 43: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/43.jpg)
Equivalent mutations
// Originalint i = 0;while (i != 10) { doSomething(); i += 1;}
// Mutantint i = 0;while (i < 10) { doSomething(); i += 1;}
![Page 44: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/44.jpg)
Infinite Runtime
// Originalwhile (expression) doSomething();
// Mutantwhile (true) doSomething();
![Page 45: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/45.jpg)
Disadvantages
● Can slow down your TDD rhythm
● May be very noisy
![Page 46: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/46.jpg)
Let’s say we have codebase with:● 300 classes● around 10 tests per class● 1 test runs around 1ms● total test suite runtime is about 3s
Is it really slow?
Let’s do 10 mutations per class● We get 3000 (300 * 10) mutations● runtime with all mutations is 150 minutes (3s * 3000)
![Page 47: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/47.jpg)
Speeding it up
Run only tests that cover the mutation● 300 classes● 10 tests per class● 10 mutations per class● 1ms test runtime● total mutation runtime 10 * 10 * 1 * 300 = 30s
![Page 48: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/48.jpg)
Speeding it up
During development run tests that cover only your current changes
![Page 49: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/49.jpg)
● Continuous integration● TDD with mutation testing only
on new changes● Add mutation testing to your
legacy project, but do not fail a build - produce warning report
Usage scenarios
![Page 50: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/50.jpg)
Tools
● Ruby - Mutant● Java - PIT● And many tools for other
languages
![Page 51: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/51.jpg)
Summary
● Code coverage highlights code that is definitely not tested
● Mutation testing highlights code that definitely is tested
● Given non equivalent mutations, good test suite should work the same as a hash function
![Page 52: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/52.jpg)
Vaidas Pilkauskas
@liucijus
● Vilnius JUG co-founder
● Vilnius Scala leader
● Coderetreat facilitator
● Mountain bicycle rider
● Snowboarder
About usTadas Ščerbinskas
@tadassce
● VilniusRB co-organizer
● RubyConfLT co-organizer
● RailsGirls Vilnius & Berlin coach
● Various board sports’ enthusiast
![Page 53: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/53.jpg)
Credits
A lot of presentation content is based on work by these guys
● Markus Schirp - author of Mutant● Henry Coles - author of PIT● Filip Van Laenen - working on a book
![Page 54: Can You Trust Your Tests? (Agile Tour 2015 Kaunas)](https://reader031.fdocuments.net/reader031/viewer/2022021923/58ed210e1a28ab31568b46bd/html5/thumbnails/54.jpg)
Q&A