Emerging Paradigms in Testing - ANZTB vendors using mixed approaches to development, agile and...
Transcript of Emerging Paradigms in Testing - ANZTB vendors using mixed approaches to development, agile and...
© ThoughtWorks 2012
Emerging Paradigms in Testing
Refocus your view of testing to drive superior results
© ThoughtWorks 2012
A good testing experience
“The quality of the application, rapidity of testing, as well as the ability to easily and confidently update the service once live, were major benefits. The test driven development and continuous integration approach was the most professional I’ve seen and provided very high quality and stability.”
Ian Frew, Leader IT, The Buzz at IAG
© ThoughtWorks 2012
The case study ! Multiple vendors using mixed approaches to development, agile and waterfall -
Approximately 60 developers ! Mainframe backend, SOA and multiple front-end systems and external 3rd
party system. ! Testing covered performance, security (including PCI), disaster recovery, deploy/
rollback, failover/recovery, user acceptance, usability and functional.
! 5 full–time testers
! After go live could make a code change and deploy in the same day no Severity 1 or Severity 2 defects in 9 months.
! 6 months development, after last feature developed 1 week to go live. no Severity 1 defects in first 9 months.
© ThoughtWorks 2012
Testing Principles
We believe testing should focus on building quality in rather than "testing it in". We do this by involving everyone in the testing effort, using fast feedback to make decisions. We believe tests are an asset of the product and not the project. Working in this way enables faster delivery into production, provides a clear and consistent view of testing across the organization and helps business optimize value within the software.
© ThoughtWorks 2012
Before we get into the details
! Describe the principle ! Present a practice demonstrating the principle in action ! Provide a real world example from the case study highlighting
the principle in action
© ThoughtWorks 2012
Principle: Build quality in
! Remove the cost of fixing the defects by preventing them from
occurring.
! Find mistakes early and prevent them from becoming prevalent.
! Reduce reliance on band aid fixes.
“You can not inspect quality into the product; it is already there.” – W. Edwards Deming
© ThoughtWorks 2012
Practice: ATDD (Acceptance Test Driven Development)
BA works with business to define a feature in a testable, bite sized vertical slice (implementable in
2-3 days)
BA and Tester work with business to define how the
feature will be tested
Tester works with developers to implement the tests and
developers implement the feature
BA and Tester get handover of feature from developers
BA and Tester test the feature and how it fits in to
the broader system and business process
X test fails
✓ Test passes
© ThoughtWorks 2012
Example: Defects as a measure
Preventing Defects
Finding Defects Preventing Defects
Finding Defects
© ThoughtWorks 2012
Principle: Fast feedback
! Get feedback to the people who need it as soon as possible.
! Work with up to date information and make informed decisions.
! Regression suites should be run in hours not weeks.
© ThoughtWorks 2012
Practice: Test Pyramid
Unit
Integration
Acceptance
Scenario
More tests Increased execution Time
© ThoughtWorks 2012
Example: Mainframe developers
http://www.kome-computers.com/image-files/mainframe.gif
© ThoughtWorks 2012
Principle: Involve everyone
! Target utilization of testing resources.
! Get better tests through diverse input and the team looking after the tests.
! Better use of resources – equalizing.
Testing ≠ Testers
© ThoughtWorks 2012
Practice: Automation Framework Write Test Specification in Text
Use Code
Write Code
BUSINESSREPORT
TECHNICALREPORT
TEST INTENTION DATA
HTML SERVICE DATABASE REPORT
DOMAIN SPECIFIC LANGUAGE (DSL)
TEST RUNNER
© ThoughtWorks 2012
Example: Rating Table
http://www.car-insurance-for-learner-driver.com/images/group-2-car-insurance-pic.jpg
© ThoughtWorks 2012
Principle: Tests as an asset
! Reuse tests across projects, don’t create new tests all of the time.
! Better understanding of test coverage.
! Treat the test cases and the application code as the product – don’t separate them.
Code
Tests Product +
© ThoughtWorks 2012
Practice: Version Control
! Automated Tests are code – treat them like code
! Test code is likely to be the largest code base in your organisation
! Version controlling application code is common practice – so it should be for test code.
© ThoughtWorks 2012
Example: Can’t rollback tests
http://images.coplusk.net/projects/10202/steps/20603/full_20603.jpg
© ThoughtWorks 2012
Principle: Faster delivery into production
! Do not have months of testing after development is complete.
! Return on investment early – start earning revenue as soon as you can.
! Testers should stop focusing on preventing a product going live and instead focus on how to get it live.
© ThoughtWorks 2012
Practice: Build Pipeline
© ThoughtWorks 2012
Example: Dev to Prod in same day
FrontEnd Dev CMS Dev Frontend Dev 3rd Party Doc Dev 3rd Party
Claim Dev
Security Dev
Claims Call Center
Call Center
Dev
Dev Test
Dev
Pre-Int Int Sys SIT UAT
DR
Prod
Dev Dev Dev
Operations
Security
Performance
Services
Acceptance
Exploratory
Integration
© ThoughtWorks 2012
Principle: Clear & consistent view
! Simple reporting so you can compare one project with another.
! Metrics should not drive behavior.
! Open and honest.
© ThoughtWorks 2012
Practice: Example Report
© ThoughtWorks 2012
Principle: Optimize business value
! Testing is not just insurance. ! Target testing to features that are changing not a scatter gun test
everything approach. ! Help drive new features and functionality. ! Allow business to try new ideas in a safe environment providing quick
feedback
© ThoughtWorks 2012
It is not just small web apps!
Data warehousing - consolidation of multiple (approx 30) data warehousing solutions into one. Multiple streams in the program of approx 150 people.
Trainline - Train ticketing system: distributed team, UK and India approximately 200 People.
Financial Services – integration of over 70 systems (internal and 3rd party). Distributed team, Australia and India approximately 100 People
© ThoughtWorks 2012
It is about the principles!