#LFMF  

Learn  from  My  Fail:    

Tales  of  Test  Automa6on  Gone  Wrong    

(Plus  a  Li*le  Bit  about  How  to  Do  It  Right)  

(and others who shall remain anonymous even if they don’t deserve it)!

Elisabeth  Hendrickson  Quality  Tree  So7ware,  Inc.  

@testobsessed  

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  

Fail  Pa;ern  #0:    “No  6me  to  automate”  No  automaFon  means  rapid  growth  in  the  

manual  tesFng  burden.  Or  untested  so7ware.  

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  

Velocity!

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  

Fail  Pa;ern  #1:    Automa6on  for  the  sake  

of  automa6on  

If  you  really  want  a  mess,  use  a  count  of  automated  tests  as  a  measure  of  

progress.  

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  

Image  of  prinFng  office  from  hJp://www.shorpy.com/node/12652  

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  

Fail  Pa;ern  #2:    All  automa6on  is  end-­‐to-­‐end  through  the  GUI  

End-­‐to-­‐end  tests  are  slow  and  fragile.  Some  are  good.  

More  are  not  beJer.  

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  

Flooded  basement  image  courtesy  of  carlpenergy  Showerhead  image  courtesy  of  deusx  Images  from  Flickr  licensed  through  CreaFve  Commons  AJribuFon  

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  

Fail  Pa;ern  #3:    Dispropor6onate  

Amount  of  Time  Spent  Fixing  Broken  Tests  Just  when  you  get  one  working  again,  another  

breaks.  

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  

Image  courtesy  of  cdrussorusso  From  hJp://www.flickr.com/photos/23516192@N08/2631474033/    Licensed  Under  CreaFve  Commons  AJribuFon  

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  

Fail  Pa;ern  #4:    Fragile  Tests  

They  pass,  they  fail,  but  there  seems  to  be  no  

correlaFon  between  the  test  results  and  the  actual  so7ware  behavior.  

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  Hey, the

build is red.!

Yeah, it’s the 2nd Tuesday with a full moon this

year.!

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  

Fail  Pa;ern  #5:    Inter-­‐dependent  Tests  

When  one  fails,    they  all  fail  together.  

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  

Run #1!

Run #2!

Run #3!

Run #4!

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  

Fail  Pa;ern  #6:    Useless  Tests  

The  tests  can  never  fail,  so  they  yield  no  informaFon.  

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  …

assert True

… But rarely. More common: tests that

verify the software

does what the software does.!

Yes, I have actually seen this in tests.!

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  

Fail  Pa;ern  #7:    Bad  Tests  

Obfuscated  intent,  massive  duplicaFon,  and  Fght  coupling  between  the  interface  and  the  

tests  makes  for  an  unmaintainable  mess.  

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  

Test Login

Input Text first_name fred

Input Text last_name flintstone

Input Text user_name fflint

Input Text password p@ssw0rd

Click Submit

Click Logout

Click Login

Input Text username fflint

Input Text password badpassword

Click Submit

Input Text ...

WAT?!?!

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  

Fail  Pa;ern  #8:    Bad  Code  

Smelly  test  automaFon  code  that  violates  SOLID  

principles  is  as  much  of  a  problem  as  smelly  producFon  code.  

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  

Fail  Pa;ern  #9:    Only  specialists  

automate  

They  might  be  good  at  automaFon,  but  they’re  fighFng  an  uphill  baJle.  

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  

Fail  Pa;ern  #10:    Forcing  devout  non-­‐

coders  to  write  automa6on  

If  they  lack  either  interest  or  apFtude  to  automate,  the  

results  will  be  bad.  

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  

Fail  Pa;ern  #11:    Tests  That  Don’t  Get  Run  

Someone  writes  them,  no  one  runs  them.  So  when  they  suffer  bitrot,  no  one  can  hear  them  scream.  

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  

Photo  by  crabchick  hJp://www.flickr.com/photos/crabchick/2627370039/  Licensed  under  CreaFve  Commons  AJribuFon  

Unrun !tests!become !creaky,!dusty,!sad.!

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  

Fail  Pa;ern  #12:    Failing  Tests  Ignored  

They  run.    

They  fail.    

No  one  cares.  

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  

Image  of  the  1906  San  Francisco  Earthquake  a7ermath  from  hJp://www.shorpy.com/node/9866  

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  #LFMF  

Automate important expectations!Treat test code with the same level of care as production code!

Run all the automated tests in the CI build (no manual kickoff)!Fix failing tests immediately!Foster a “whole team tests” culture!

Copyright  ©

 2012  Elisabeth  Hen

drickson

,  Quality  Tree  So7

ware,  Inc.  

Many thanks to …!@hackyourtv!Juha Rantanen!Jyrki Pulliainen!Jarkko Kailanto!Teema Frisk!