Post on 16-May-2015
‘09
Madrid, 27th-29th of October 2009
Igor GershovichConnected Testing, Inc.www.connectedtesting.com
‘09Madrid, 27th-29th of October 2009
About Connected Testing, Inc
• Connected Testing is an independent consulting company specializing in Test Automation and Performance Testing
• We are located in Denver, Colorado, USA• We have extensive experience across a range of
industries including Financial Services, Healthcare, Aerospace, Transportation, Telecommunications
‘09Madrid, 27th-29th of October 2009
Overview
• Test Automation Myths
• Return on Investment (ROI)
• Frameworks
‘09Madrid, 27th-29th of October 2009
Test Automation Myths
‘09Madrid, 27th-29th of October 2009
Myth #1 - Test Automation is simple, that every tester can do it
• This myth is promoted by the tool sales people. They are trying to promote the following test automation process:– Record the script– Enhance the script by adding functions and data
driving– Run the scripts– Report results
• Under the influence of this myth the QA manager can proudly report: All our testers are developing test automation.
‘09Madrid, 27th-29th of October 2009
Reality - Test automation is a software development task
• Automation should be designed, developed and tested
• You need to have some kind of a programming background to implement test automation. Test Automation is not as complex as C++/C#/Java development.
• Test automation standards should be developed• Automated test components are assets that
should be treated like application source code
‘09Madrid, 27th-29th of October 2009
Myth #2 – Commercial test tools are expensive
• Under the influence of this myth some companies, especially the small ones:– Try to develop their own test automation tools– Use scripting languages like Perl and Ruby– Use shareware test tools– Do not consider test automation at all
‘09Madrid, 27th-29th of October 2009
Reality – Commercial tools are cheap
• Per seat license for the most expensive automation tool is $8K
• This tool will be used for 5 years.• Maintenance/Support fees are 20% of tool cost or
$1,800 per year• The cost of this tool is $8K/5+$1,800 = $3,100 per year• The automation developer cost with overhead is $100K
per year• The cost of this tool is just 3% of the person who uses it,
but productivity gain can be very significant
‘09Madrid, 27th-29th of October 2009
Commercial Tool Benefits:• Customer support. Many of the open source tools come and go with
little to no support • Most commercial tools are constantly being updated as
technologies change• Most commercial tools usually have more functionality (QTP can
test various GUI applications: Web, .Net, Java, VB, C/C++, PowerBuilder, etc. vs. WATIR – Web only)
• Commercial tools usually have a large community of users, which translates into better availability of qualified resources
• Commercial tools require less advanced programming• More test automation frameworks are available for commercial tools• Commercial tools are integrated with Test Management tools which
makes reporting and execution much simpler.
‘09Madrid, 27th-29th of October 2009
Over 300 Test Tools are available (http://www.softwareqatest.com)
• Load/Performance tools – 54
• Web Functional/Regression – 60
• Java Test tools - 48
• Other Web tools – 76
Which tool is right for you?
‘09Madrid, 27th-29th of October 2009
Some Requirements for Tool evaluation
• Technologies that should be supported• Available training and documentation• Cost• Support• Distributed execution• Non-proprietary language• Tool Audience (The skills of people involved in test
automation)• Easy to maintain automated tests with a central
repository
‘09Madrid, 27th-29th of October 2009
Main Reasons for Test Automation failures
• Lack of Management Support
• Inexperienced Test Automation staff
• Over-engineering
‘09Madrid, 27th-29th of October 2009
Think of yourself like a business
• Facts:– Average IT job tenure is about 2 years– Competitive intensity is rising
• What you should do:– Invest your time/money in learning mainstream
automation– Create a niche for yourself - add Configuration
Management, Performance Testing or Project Management to your skillset
– Market yourself
‘09Madrid, 27th-29th of October 2009
Absolutely Unscientific research:
Employer demand by tool expertiseMay, 2008 and Aug, 2009
‘09Madrid, 27th-29th of October 2009
Dice.com search results across US (30 days)
Tool Search string Matches
May, 08
Matches
Aug, 09
QTP Qtp OR quicktest OR "quick test“
613 484
Functional Tester
"rational robot" OR "functional tester”
118 58
SilkTest silktest OR "silk test” 86 31
TestPartner testpartner OR "test partner“ 19 18
WATIR watir OR watiN or WatiJ 26 28
Selenium Selenium 108 140
Total 22% decline from May, 08
to Aug, 09
970 759
‘09Madrid, 27th-29th of October 2009
HP QTP64%
IBM Functional Tester
8%
Borland SilkTest
4%
CompuwareTestPartner
2%
WATIR4%
Selenium18%
Demand per tool - August 2009
QTP
Functional Tester
SilkTest
TestPartner
WATIR
Selenium
‘09Madrid, 27th-29th of October 2009
IDC Report: “Worldwide Automated Software Quality 2007-2011 Forecast and 2006 Vendor Shares…”
‘09Madrid, 27th-29th of October 2009
Return On Investment (ROI)
‘09Madrid, 27th-29th of October 2009
Classic ROI Calculation
ROI = BENEFIT/COST
Automation Cost = Price Of HW + Price of SW + Development Cost + Maintenance Cost + Execution Cost
Manual Testing Cost = Development Cost + Maintenance Cost + Execution Cost
ROI = (Manual Testing Cost - Automation Cost)/Automation Cost
Looks right, Doesn’t it?
‘09Madrid, 27th-29th of October 2009
Problems with Classic ROI Calculation
You can’t compare Automated Testing and Manual Testing. They are not the same and they provide different information about the AUT.
You can’t compare cost of multiple execution of automated tests vs. manual tests. You would never dream of executing that many test cases manually.
‘09Madrid, 27th-29th of October 2009
Automation Real ROI
ROI value IS NOT the value of Automation vs. Cost of executing these tests manually
Automation ROI value IS the benefit of this type of testing, and it can be:Reducing Time to MarketIncreased Test Efficiency (Productivity)Increased Test Effectiveness
‘09Madrid, 27th-29th of October 2009
Reduced Time to Market
• Can get a greater market share
• Makes people available to work on other projects
• Higher margins, if no competitive products are currently available
‘09Madrid, 27th-29th of October 2009
Productivity and Effectiveness• More testing gets done faster, increasing the
odds of finding defects• Defects found early have better chances of
being fixed• Manual Testers can concentrate on clever ways
to finding defects, instead of typing test inputs and verify output.
• About 7% of bug fixes create new bugs, sometimes in already tested parts of the system. With automation you can rerun tests for those modules. This almost never happens when testing is done manually.
‘09Madrid, 27th-29th of October 2009
Example: 2-week Regression
1st week1st week 2nd week2nd week
Manual Testing Bug Fixes/New Build
Spot-check
Manual Regression
1st week1st week 2nd week2nd week
Automated and ManualTesting – 1st Run
Automated Regression
Bug Fixes/New Build
Automated and ManualTesting – 2nd Run
‘09Madrid, 27th-29th of October 2009
Advantages of automated regression testing
• 2 full regression runs
• Most of bugs were found early and have a better chance to be fixed
Result:– Better quality– much happier customer
‘09Madrid, 27th-29th of October 2009
ROI summary
• Each project requires different types of automation - there is often no easy formula available to calculate ROI
• Performing ROI calculation can help to determine upfront what type of automation, what level of skills, what tools will be required.
‘09Madrid, 27th-29th of October 2009
Automation Frameworks
‘09Madrid, 27th-29th of October 2009
Test Automation Framework
• A Test Automation Framework is a set of assumptions, concepts and tools that provide support for Automated Software Testing.
• A correctly implemented Test Automation Framework can further improve ROI by reducing the development and maintenance costs.
‘09Madrid, 27th-29th of October 2009
Types of Test Automation Frameworks
• Modular
• Data-Driven
• Keyword–Driven
• Model-Based
‘09Madrid, 27th-29th of October 2009
Modular framework
• The Modular framework is the natural progression derived from Record-and-Playback
• The modular framework seeks to minimize repetition of code by grouping similar actions into “modules” (e.g.: login)
• Test Data is in a script or in an internal table
This is what tool sales people are promoting.
‘09Madrid, 27th-29th of October 2009
Data-Driven/Keyword Driven• These frameworks are similar in that the
data is separated from the test script• The script is just a "driver" or delivery
mechanism for the data.• The difference:
– In keyword-driven testing, the navigation data and test data are contained in the data source
– In data-driven testing, only test data is contained in the data source.
‘09Madrid, 27th-29th of October 2009
Model-Based testing
• Model-based testing is software testing in which test cases are derived in whole or in part from a model that describes some (usually functional) aspects of the system.
• Model-based testing for complex software systems is still an evolving field.
‘09Madrid, 27th-29th of October 2009
We’ll be concentrating on Keyword-Driven testing since it is the most beneficial framework for large-scale test automation.
‘09Madrid, 27th-29th of October 2009
Advantages of Keyword-Driven Test automation
• This Framework addresses the most common problem with test automation: Automation Engineers do not have domain knowledge and the End Users (Subject Matter Experts/Test Engineers) usually do not have automation expertise.
• When properly implemented and maintained, it presents a superior ROI because each business event is designed, automated and maintained as a discrete entity.
• Keywords can then be used to design test cases, but the design and automation overhead for the keyword has already been paid.
‘09Madrid, 27th-29th of October 2009
Advantages of Keyword-Driven Test automation
• Reduced the cost and time spent maintaining and updating tests
• The modular structure of keyword-driven testing means that new tests can easily be created from pre-existing modules
• The test team is capable of entirely automating tests, even without programming knowledge
• Can be easily modified to use with different test tool
• Reusability across different projects
‘09Madrid, 27th-29th of October 2009
Classic Keyword-Driven Example
Object Action Data
Textfield (username) Enter Text <username>
‘09Madrid, 27th-29th of October 2009
Example of Test Data for Keyword-Driven test automation developed by Connected Testing Inc.
for TestComplete and QTP
‘09Madrid, 27th-29th of October 2009
Resources• “Everything you want to know about Test Automation…” by Brian Le
Suer• “Manager’s Guide to GUI Test Automation” by Yury Makedonov• “The ROI of Test Automation” by Michael Kelly• “What is my ROI?” by mVerify Corporation• SQA Forums• “An Overview of Test Automation Frameworks” by Nathaniel
Ritmeyer• “The benefits of keyword-based software test automation” by David
W. Johnson • Wikipedia
‘09Madrid, 27th-29th of October 2009
Igor Gershovich
Company: Connected Testing, Inc.
E-Mail: igershovich@connectedtesting.com
Phone: 720-933-9395
Website: www.connectedtesting.com