The Lies We Tell About Software Testing

59
Tes$ng Lies (The &tle is the first lie) Hannover, Germany, 2014 Cur$s “Ovid” Poe h5p://www.allaroundtheworld.fr/ Photo CC license: h5p://commons.wikimedia.org/wiki/File:Hannover_Altstadt_128h.jpg Copyright 2014, All Around The World March 26, 2014

description

This talk is more about half-truths than lies, but the concepts are important. There are several areas in which we think we have more confidence in our information than we actually do. This presentation covers both areas in which the evidence is against our beliefs and areas in which we simply need more information.

Transcript of The Lies We Tell About Software Testing

Page 1: The Lies We Tell About Software Testing

Tes$ng  Lies  

(The  &tle  is  the  first  lie)  Hannover,  Germany,  2014  

Cur$s  “Ovid”  Poe  h5p://www.allaroundtheworld.fr/  

Photo  CC  license:  h5p://commons.wikimedia.org/wiki/File:Hannover_Altstadt_128-­‐h.jpg  

Copyright  2014,  All  Around  The  World  March  26,  2014  

Page 2: The Lies We Tell About Software Testing

But  first  …  

March  26,  2014   Copyright  2014,  All  Around  The  World  

h5p://www.slideshare.net/gvwilson/bits-­‐of-­‐evidence-­‐2338367  

Page 3: The Lies We Tell About Software Testing

Bits  of  Evidence  by  Dr.  Greg  Wilson  

March  26,  2014   Copyright  2014,  All  Around  The  World  

•  Overwhelming  evidence  that  evidence  works  •  Science  demands  evidence  

•  Programmers  provide  anecdotes  

•  (…  and  defend  them  vigorously)  

Page 4: The Lies We Tell About Software Testing

Evidence  

•  Absence  of  evidence  is  not  evidence  •  …  but  evidence  of  absence  is  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 5: The Lies We Tell About Software Testing

Absence  of  Evidence  

•  There  is  no  evidence  that  gods  exist  •  There  is  no  evidence  that  gods  don’t  exist  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 6: The Lies We Tell About Software Testing

Evidence  of  Absence  

•  Belief:  small  subs  more  maintainable/correct  •  Reality:  these  proper$es  are  absent  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 7: The Lies We Tell About Software Testing

Evidence  of  Absence  

•  Belief:  small  subrou$nes/methods  “be5er”  •  Inverse  correla$on  (Bell,  Perricone  1984)  •  No  correla$on  (Shen,  et  al.  1985)  •  No  correla$on  and  larger  is  cheaper  (two  studies)  •  Small  rou$nes  had  more  errors  (Selbi,  Basil  1991)  •  Less  maintenance  for  larger  rou$nes  (Lind,  Vairavan  1989)  

•  500+  lines  had  more  errors  (Jones  1986)  

(cita$ons  via:  Code  Complete,  Steve  McConnell,  Microsoh  Press,  2nd  Edi$on)  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 8: The Lies We Tell About Software Testing

P.O.P.  Copyright  2013,  Cur$s  "Ovid"  Poe  

Reality  trumps  opinion  

3/26/14  

Page 9: The Lies We Tell About Software Testing

Yeah,  yeah,  let’s  get  started  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 10: The Lies We Tell About Software Testing

Caveats  

March  26,  2014   Copyright  2014,  All  Around  The  World  

•  Some  points  self-­‐evident  •  Some  points  maddening  

•  Not  many  studies  

•  Always  be  skep$cal  

Photo  CC  BY-­‐SA  2.0  h5p://www.flickr.com/photos/busbeytheelder/515542705/  

Page 11: The Lies We Tell About Software Testing

Code  Coverage  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 12: The Lies We Tell About Software Testing

100%  Code  Coverage  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 13: The Lies We Tell About Software Testing

100%  Code  Coverage  

•  What  if  we  pass  zero?  •  …  or  a  string?  •  …  or  no  argument?  

•  …  or  a  filehandle?  •  …  or  type  when  reciprocals  don't  make  sense?  

•  …  or,  or,  or  …  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 14: The Lies We Tell About Software Testing

Expecta$ons  

•  Tes$ng  should  be  an  act  of  aggression  •  Programmers  tend  to  test  against  expecta$ons  

•  False  confidence  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 15: The Lies We Tell About Software Testing

Code  Coverage  of  Dead  Code  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 16: The Lies We Tell About Software Testing

Unit  Versus  Integra$on  Tests  

•  Unit  tested  subs  don’t  talk  to  other  subs  •  But  your  code  certainly  does!  •  Dead  code  is  cogni$ve  overhead  •  Dead  code  can’t  be  tested  via  your  interface  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 17: The Lies We Tell About Software Testing

Code  Coverage  of  Dead  Code  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 18: The Lies We Tell About Software Testing

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 19: The Lies We Tell About Software Testing

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 20: The Lies We Tell About Software Testing

Covered  but  not  tested  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 21: The Lies We Tell About Software Testing

Devel::Cover  Recap  

•  100%  code  coverage  doesn’t  handle  data  •  Don’t  cover  dead  code  •  Covered  but  not  tested  •  Don't  blindly  trust  your  tools  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 22: The Lies We Tell About Software Testing

Kwalitee  

•  Lie:  Perl  community  is  the  most  test-­‐infected  •  Truth:  Quan$ty  ain’t  quality  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 23: The Lies We Tell About Software Testing

CPAN  Testers  (March  6,  2014)  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 24: The Lies We Tell About Software Testing

Versions  and  Plauorms  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 25: The Lies We Tell About Software Testing

CPAN  Testers  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 26: The Lies We Tell About Software Testing

Perl  Isn’t  Stealing  Enough  

We  need  quality  to  go  with  quan$ty  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 27: The Lies We Tell About Software Testing

All  Pairs  Tes$ng  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 28: The Lies We Tell About Software Testing

The  Coupling  Effect  

•  Inves$ga$ons  of  the  Sohware  Tes$ng  Coupling  Effect  —  A.  Jefferson  Offu5  

•  Competent  programmers  write  nearly  correct  programs  

•  Most  faults  are  simple  faults  

•  Simple  faults  can  be  easily  detected  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 29: The Lies We Tell About Software Testing

All-­‐Pairs  Tes$ng  

•  10  arguments  ==  10  billion  combina$ons  •  Combined  pairs  ==  54  combina$ons  

•  Extend  each  argument  with  reasonable  values  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 30: The Lies We Tell About Software Testing

All-­‐Pairs  Tes$ng  

First  pairs   Second  Pairs   Third  Pairs   Fourth  Pairs   And  so  on  

arg1,  arg2   —   —   —   …  

arg1,  arg3   arg2,  arg3   —   —   …  

arg1,  arg4   arg2,  arg4   arg3,  arg4   —   …  

…   …   …   arg4,  arg5   …  

arg1,  arg0   arg2,  arg0   arg3,  arg0   …   …  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 31: The Lies We Tell About Software Testing

All-­‐Pairs  Tes$ng  

March  26,  2014   Copyright  2014,  All  Around  The  World  

•  Fast  genera$on  of  tests  •  Likely  to  catch  many  more  bugs  

•  Built  in  to  many  tools  (for  sta$c  languages)  

•  For  Perl:  ALLPAIRS  Test  Case  Genera$on  Tool  

Page 32: The Lies We Tell About Software Testing

Documenta$on  

•  Lie:  Tests  are  documenta$on  •  Truth:  I  wish  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 33: The Lies We Tell About Software Testing

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 34: The Lies We Tell About Software Testing

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 35: The Lies We Tell About Software Testing

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 36: The Lies We Tell About Software Testing

Tests  As  Documenta$on  

•  Tests  are  ohen  obfuscated  •  Ohen  wri5en  as  an  aherthought  •  Missing  or  non-­‐descrip$ve  test  names  

•  Describe  code  logic  instead  of  business  logic  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 37: The Lies We Tell About Software Testing

Making  Tests  Documenta$on  

•  Write  them  to  be  read  •  Describe  use  cases  with:  –  test  fixtures  names  

– subtest  names  –  test  method  names  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 38: The Lies We Tell About Software Testing

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 39: The Lies We Tell About Software Testing

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 40: The Lies We Tell About Software Testing

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 41: The Lies We Tell About Software Testing

BDD  for  Test  Documentaiton  

•  Test::BDD::Cucumber  •  Perl  Behavior  Driven  Development  (slideshare)  

March  26,  2014   Copyright  2014,  All  Around  The  World  

h5p://commons.wikimedia.org/wiki/File:Og%C3%B3rki...jpg  

Page 42: The Lies We Tell About Software Testing

Test  Driven  Development  

•  Lie:  TDD  is  a  far  superior  tes$ng  methodology  •  Truth:  Prove  it.  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 43: The Lies We Tell About Software Testing

A  Typical  TDD  Study  

•  Evalua$ng  the  Efficacy  of  Test-­‐Driven  Development:  Industrial  Case  Studies  

•  Compared  to  non-­‐TDD  

•  TDD  increased  development  $me  ~15%  

•  TDD  had  far  fewer  defects  than  non-­‐TDD  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 44: The Lies We Tell About Software Testing

What  is  “non-­‐TDD”?  

•  No  tes$ng?  •  Test  last?  •  Test  as  you  go?  •  QA-­‐only  tes$ng?  •  It  doesn’t  say!  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 45: The Lies We Tell About Software Testing

Same  Study  

“An  (sic)  year-­‐long  empirical  study  performed  at  IBM  using  professional  programmers  found  that  the  TDD  prac?ce  helps  programmers  produce  higher  quality  code.”  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 46: The Lies We Tell About Software Testing

Same  Study  

“An  (sic)  year-­‐long  empirical  study  performed  at  IBM  using  professional  programmers  found  that  the  TDD  prac?ce  helps  programmers  produce  higher  quality  code.”  

Higher  quality  code  than  what?  

And  what  does  “quality”  mean?  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 47: The Lies We Tell About Software Testing

Pro-­‐TDD  Studies  

•  Comparisons  against  straw  men  common  •  Ohen  use  students  •  Some$mes  use  single  developers  

•  Some$mes  aware  they’re  being  studied  

•  Or  new  to  TDD  (may  be  more  mo$vated)  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 48: The Lies We Tell About Software Testing

A  TDD  Enthusiast  Looks  At  Evidence  

•  The  benefits  of  TDD  are  neither  clear  nor  are  they  apparent  —  Kane  Mar  – Part  1  – Part  2  – Part  3  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 49: The Lies We Tell About Software Testing

Kane  Mar  –  Aher  Reading  Studies  

“[Claims  about  TDD]  such  as  beHer  design,  beHer  APIs,  simpler  design,  lower  complexity,  increased  produc?vity,  more  maintainable  code  etc.,  are  simply  not  supported.”1  

1.  A  cherry-­‐picked,  out-­‐of-­‐context  quote.  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 50: The Lies We Tell About Software Testing

The  Reality  (vis-­‐à-­‐vis  other  tes$ng  strategies)  

•  TDD  tends  to  generate  more  tests  •  Higher  quality  not  proven  •  Absence  of  evidence,  not  evidence  of  absence  •  Don’t  stress  about  it  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 51: The Lies We Tell About Software Testing

Summary  

•  Evidence  trumps  anecdotes  •  We  rely  on  anecdotes  

•  That’s  sort  of  OK  if  we  don’t  have  evidence˜  •  Code  coverage  is  only  part  of  the  solu$on  •  We  need  to  escape  our  bubble  

•  Be  skep$cal  of  hype  •  Be  skep$cal  of  me  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 52: The Lies We Tell About Software Testing

Summary  for  TDD  Fana$cs  

•  Don't  sweat  anecdotes  for  casual  ideas  •  Don’t  accept  anecdotes  from  dogma$sts  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 53: The Lies We Tell About Software Testing

March  26,  2014   Copyright  2014,  All  Around  The  World  

Image  by  Hay  Kranen  

Page 54: The Lies We Tell About Software Testing

Bonus  Slides!  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 55: The Lies We Tell About Software Testing

Basis  Path  Tes$ng  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 56: The Lies We Tell About Software Testing

Cycloma$c  Complexity  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Cycloma$c  Complexity  (McCabe)  

=  (edges  –  nodes)  +  (  2  *  exit  nodes  )  =  (  19            –  14            )  +    (  2  *  1  )  =  7  =  minimum  number  tests  

Page 57: The Lies We Tell About Software Testing

Calcula$ng  Cycloma$c  Complexity  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 58: The Lies We Tell About Software Testing

Cycloma$c  Complexity  and  Bugs  

•  Some  studies  show  a  posi$ve  correla$on  •  Other  studies  are  not  conclusive  •  May  make  bugs  harder  to  fix  

•  May  lead  to  side  effects  when  making  changes  

•  A  Complexity  Measure  —  McCabe  •  A  Cri$que  of  Sohware  Defect  Predic$on  Models  —  Fenton,  Neil    

•  Cycloma$c  Complexity  Metrics  Revisited  —  Gill,  Kemerer  

March  26,  2014   Copyright  2014,  All  Around  The  World  

Page 59: The Lies We Tell About Software Testing

Perl  

March  26,  2014   Copyright  2014,  All  Around  The  World