The Lies We Tell About Software Testing

Tes$ng Lies

(The &tle is the first lie) Hannover, Germany, 2014

Cur$s “Ovid” Poe h5p://www.allaroundtheworld.fr/

Photo CC license: h5p://commons.wikimedia.org/wiki/File:Hannover_Altstadt_128-‐h.jpg

But first …

h5p://www.slideshare.net/gvwilson/bits-‐of-‐evidence-‐2338367

Bits of Evidence by Dr. Greg Wilson

•  Overwhelming evidence that evidence works •  Science demands evidence

•  Programmers provide anecdotes

•  (… and defend them vigorously)

Evidence

•  Absence of evidence is not evidence •  … but evidence of absence is

Absence of Evidence

•  There is no evidence that gods exist •  There is no evidence that gods don’t exist

Evidence of Absence

•  Belief: small subs more maintainable/correct •  Reality: these proper$es are absent

Evidence of Absence

•  Belief: small subrou$nes/methods “be5er” •  Inverse correla$on (Bell, Perricone 1984) •  No correla$on (Shen, et al. 1985) •  No correla$on and larger is cheaper (two studies) •  Small rou$nes had more errors (Selbi, Basil 1991) •  Less maintenance for larger rou$nes (Lind, Vairavan 1989)

•  500+ lines had more errors (Jones 1986)

(cita$ons via: Code Complete, Steve McConnell, Microsoh Press, 2nd Edi$on)

Reality trumps opinion

3/26/14

Yeah, yeah, let’s get started

Caveats

•  Some points self-‐evident •  Some points maddening

•  Not many studies

•  Always be skep$cal

Photo CC BY-‐SA 2.0 h5p://www.flickr.com/photos/busbeytheelder/515542705/

Code Coverage

100% Code Coverage

•  What if we pass zero? •  … or a string? •  … or no argument?

•  … or a filehandle? •  … or type when reciprocals don't make sense?

•  … or, or, or …

Expecta$ons

•  Tes$ng should be an act of aggression •  Programmers tend to test against expecta$ons

•  False confidence

Code Coverage of Dead Code

Unit Versus Integra$on Tests

•  Unit tested subs don’t talk to other subs •  But your code certainly does! •  Dead code is cogni$ve overhead •  Dead code can’t be tested via your interface

Code Coverage of Dead Code

Covered but not tested

Devel::Cover Recap

•  100% code coverage doesn’t handle data •  Don’t cover dead code •  Covered but not tested •  Don't blindly trust your tools

Kwalitee

•  Lie: Perl community is the most test-‐infected •  Truth: Quan$ty ain’t quality

CPAN Testers (March 6, 2014)

Versions and Plauorms

CPAN Testers

Perl Isn’t Stealing Enough

We need quality to go with quan$ty

All Pairs Tes$ng

The Coupling Effect

•  Inves$ga$ons of the Sohware Tes$ng Coupling Effect — A. Jefferson Offu5

•  Competent programmers write nearly correct programs

•  Most faults are simple faults

•  Simple faults can be easily detected

All-‐Pairs Tes$ng

•  10 arguments == 10 billion combina$ons •  Combined pairs == 54 combina$ons

•  Extend each argument with reasonable values

All-‐Pairs Tes$ng

First pairs Second Pairs Third Pairs Fourth Pairs And so on

arg1, arg2 — — — …

arg1, arg3 arg2, arg3 — — …

arg1, arg4 arg2, arg4 arg3, arg4 — …

… … … arg4, arg5 …

arg1, arg0 arg2, arg0 arg3, arg0 … …

All-‐Pairs Tes$ng

•  Fast genera$on of tests •  Likely to catch many more bugs

•  Built in to many tools (for sta$c languages)

•  For Perl: ALLPAIRS Test Case Genera$on Tool

Documenta$on

•  Lie: Tests are documenta$on •  Truth: I wish

Tests As Documenta$on

•  Tests are ohen obfuscated •  Ohen wri5en as an aherthought •  Missing or non-‐descrip$ve test names

•  Describe code logic instead of business logic

Making Tests Documenta$on

•  Write them to be read •  Describe use cases with: –  test fixtures names

– subtest names –  test method names

BDD for Test Documentaiton

•  Test::BDD::Cucumber •  Perl Behavior Driven Development (slideshare)

h5p://commons.wikimedia.org/wiki/File:Og%C3%B3rki...jpg

Test Driven Development

•  Lie: TDD is a far superior tes$ng methodology •  Truth: Prove it.

A Typical TDD Study

•  Evalua$ng the Efficacy of Test-‐Driven Development: Industrial Case Studies

•  Compared to non-‐TDD

•  TDD increased development $me ~15%

•  TDD had far fewer defects than non-‐TDD

What is “non-‐TDD”?

•  No tes$ng? •  Test last? •  Test as you go? •  QA-‐only tes$ng? •  It doesn’t say!

Same Study

“An (sic) year-‐long empirical study performed at IBM using professional programmers found that the TDD prac?ce helps programmers produce higher quality code.”

Same Study

“An (sic) year-‐long empirical study performed at IBM using professional programmers found that the TDD prac?ce helps programmers produce higher quality code.”

Higher quality code than what?

And what does “quality” mean?

Pro-‐TDD Studies

•  Comparisons against straw men common •  Ohen use students •  Some$mes use single developers

•  Some$mes aware they’re being studied

•  Or new to TDD (may be more mo$vated)

A TDD Enthusiast Looks At Evidence

•  The benefits of TDD are neither clear nor are they apparent — Kane Mar – Part 1 – Part 2 – Part 3

Kane Mar – Aher Reading Studies

“[Claims about TDD] such as beHer design, beHer APIs, simpler design, lower complexity, increased produc?vity, more maintainable code etc., are simply not supported.”1

1. A cherry-‐picked, out-‐of-‐context quote.

The Reality (vis-‐à-‐vis other tes$ng strategies)

•  TDD tends to generate more tests •  Higher quality not proven •  Absence of evidence, not evidence of absence •  Don’t stress about it

Summary

•  Evidence trumps anecdotes •  We rely on anecdotes

•  That’s sort of OK if we don’t have evidence˜ •  Code coverage is only part of the solu$on •  We need to escape our bubble

•  Be skep$cal of hype •  Be skep$cal of me

Summary for TDD Fana$cs

•  Don't sweat anecdotes for casual ideas •  Don’t accept anecdotes from dogma$sts

Image by Hay Kranen

Bonus Slides!

Basis Path Tes$ng

Cycloma$c Complexity

Cycloma$c Complexity (McCabe)

= (edges – nodes) + ( 2 * exit nodes ) = ( 19 – 14 ) + ( 2 * 1 ) = 7 = minimum number tests

Calcula$ng Cycloma$c Complexity

Cycloma$c Complexity and Bugs

•  Some studies show a posi$ve correla$on •  Other studies are not conclusive •  May make bugs harder to fix

•  May lead to side effects when making changes

•  A Complexity Measure — McCabe •  A Cri$que of Sohware Defect Predic$on Models — Fenton, Neil

•  Cycloma$c Complexity Metrics Revisited — Gill, Kemerer

The Lies We Tell About Software Testing

Technology

Transcript of The Lies We Tell About Software Testing

Detecting correcting lies that data tell - Chewy Chunks

Lies CMIOs Tell- Dr. David Allard, Henry Ford Health System

Show & tell - Summary section user testing

11 damn lies we tell our kids (about maths)

Here Are Some Lies People Tell You About Infant Sleep _ Sweet Madeleine

Direct-to-Consumer Testing: The Business with Lifestyle Tests€¦ · Bad Blood: Secrets and Lies “They basically tell the story of how Elizabeth Holmes created these fraudulent

Tell Me Lies: Propaganda and Media Distortion in the Attack on Iraq - David Miller (Editor)

“the lies we tell our code” @misterbissonevents17.linuxfoundation.org/sites/events/files/slides/The lies we tell our code... · Virtual memory according to Poul-Henning Kamp Take

“the lies we tell our code” @misterbisson lies... · Node.js enterprise support Best Practices Performance Analysis Core File Analysis Debugging Support Critical Incident Support

10 Lies Auto Insurance Companies Tell You After a Car Accident

1 Deceptive Speech CS4706 Julia Hirschberg 2 Everyday Lies Ordinary people tell an average of 2 lies per day –Your new hair-cut looks great. –I’m sorry.

Top 10 White Lies Managers Tell

LOL_ 20 Lies Designers Tell Their Clients

“What Does Exercise Testing Tell Us About Congenital Heart ...

The lies we tell ourselves - Welcome to Ekklesia | Ekklesia

ASK ME NO QUESTIONS I LL TELL YOU O LIES DOES THE … · Ardoin, Phillip and Ronald J Vogel. (2008) Ask Me No Question, I'll Tell You No Lies: Does the Bradley Effect Still Exist?

A/B Testing - What your mother didn't tell you

The Biggest Lies That Digital Marketers Tell Themselves - 3XE Digital

The Dead Never Tell Lies… A Lower Sec Science Enrichment 1.

The 10 Lies We Tell Ourselves _ Body, Mind, Soul & Spirit – UPDATED DAILY! _ BodyMindSoulSpirit