Automated  Tes,ng,  Con,nuous  Integra,on  &  Con,nuous  Deployment  

William  Josephson  Co-­‐Founder,  Solano  Labs  @SolanoLabs    Automated  Tes,ng  NYC  

Some Metrics on how Fast we have Sped Up our Customers:

•  18 hrs serial, 2 + hrs on Parallel Team City w/ local HW to 13 mins •  30 min on Jenkins to 6 mins •  5 hrs serial PyPy test suite to 15 mins •  5 hour queuing to 5 min max wait time

Solano Labs – Lets you Test More, Faster

2 Solano Labs © 2013 Confidential

Some Facts & Figures*

High performing organizations:

•  Deploy code 30 times more frequently.

–  8000 times faster than their peers

–  Deploy multiple times a day, versus an average of once a month.

•  Have 50 percent fewer failures.

•  Restore service 12 times faster than their peers.

–  Fewer failures and faster recovery mean less risk to the business when

changes are deployed.

*Source:  Puppet  Labs  2013  State  of  DevOps  Report  

How  are  they  doing  this?  

I  don’t  have  all  the  answers…  

But  here’s  one:    Automa,on  

Automated Testing Does More Than Find Bugs Its Helps Teams Build a Better Product

Bug are Common, Difficult to Eliminate & Very Costly

Automated Testing: Using software to test software

•  Reduces  cost  of  bugs  by  reducing  ,me  between  introduc,on  and  discovery  

•  Frees  engineers  to  make  large  changes  fearlessly,  reducing  cycle  ,me  

•  Reduces  complexity  and  risk  at  feature  integra,on  

Automation: Reduces the Cognitive Feedback Loop

Automa,on  shortens  the  cogni,ve  loop  for  engineers        •  Automa,on  delivers  results  for  a  build  quickly  •  Shorter  turn-­‐around  ,mes  change  how  

engineers  work  •  Test-­‐Driven  Development  (TDD)  is  one  logical  

extreme  

Waiting for tests to run is no fun!

Incorporate  Feedback  

Run  Tests  

Write  Code  

Get  Results  

What Makes for a Good Automated Test?

•  Eliminate repetitive manual tests

•  Yield reproducible results

•  Cover common and edge/error cases

•  Ideally either pure unit or integration test

•  Strive for clarity even when code is gnarly

•  Test for business requirements

Why Not Automate Everything?

•  Writing tests does have a cost

–  May not be appropriate for experimental code

–  May be fragile – UI/mobile automated testing is evolving

–  Difficult or impossible to express design and human factors

•  Thorough testing requires commitment and discipline

–  Writing tests is a short-term pain for a specific individual

–  Test suite as a whole is valuable to the team over time

What  do  you  do  with  all  those    automated  tests?  

Continuous Integration: A Discipline for Testing Every Change

I              Strong                  Isola,on  

             Detailed                Instrumenta,on  

           Consistent          Run,me        Environment        

             Reproducible                Results  

Components that Make for Effective CI:

Typical CI Setup:

Continuous Deployment: Automatically Release Validated Changes

•  Goal: make release a non-event –  Popularized by Web 2.0, but much more widely applicable –  Final decision to deploy may still be manual

•  Valuable to business and engineering sides of the house: –  Selectively enabled features – feature flags –  Consistent, widespread measurement

•  Key Ingredients: –  Small changes, continuously integrated –  Automated infrastructure (DevOps) –  Staging/test environment accurately models production

Further Reading

•  Martin Fowler from ThoughtWorks: martinfowler.com

•  Book: Continuous Delivery: Reliable Software (Jez Humble)

•  IMVU: http://timothyfitz.com/2009/02/10/continuous-deployment-at-

imvu-doing-the-impossible-fifty-times-a-day/

•  Etsy Blog: codeascraft.com

•  Github: teach.github.com/articles/lesson-continuous-delivery/

•  http://info.puppetlabs.com/2013-state-of-devops-report.html

•  http://engineering.quora.com/Continuous-Deployment-at-Quora