Software test automation_overview
-
Upload
rohan-bhattarai -
Category
Documents
-
view
876 -
download
0
description
Transcript of Software test automation_overview
![Page 1: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/1.jpg)
Software Test Automation Overview
Rohan Bhattarai
![Page 2: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/2.jpg)
Contents Overview of STA BEP and ROI Tools Automation
Framework
![Page 3: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/3.jpg)
Contents Overview of STA
History Myths and Truths Why Automation Projects Fail? What is TA? Why TA is needed? Manual vs. Automated Pros & Cons What to Automate? What Not to Automate?
BEP and ROI BEP BEP - Example ROI Classic ROI Real ROI Benefits of ROI ROI Calculator
Tools Choosing the right tool – The Strategic
Approach Choosing the right tool – The Strategic
Steps Tools Classification Short List Tools – Functional Short List Tools – Performance Evaluate Vendors Functional Test Tools - Analysis Feasibility Analysis
Automation Framework Where to start? What is AF? Benefits of AF How does it work? Architecture Framework Approaches
Record and Playback Script Based Approach Keyword Driven Approach Data Driven Approach Hybrid Keyword and Data Driven Approach
![Page 4: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/4.jpg)
Contents Overview
History Myths and Truths Why Automation Projects Fail? What is TA? Why TA is needed? Manual vs. Automated Pros & Cons What to Automate? What Not to Automate?
![Page 5: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/5.jpg)
History
Moving swiftly past the hype
Historically Automation is perceived as a “Silver Bullet” of the Testing world
“The term has been adopted into a general metaphor, where “silver bullet” refers to any straight forward solution perceived to have extreme effectiveness. The phrase typically appears with an expectation that some new technology or practice will easily cure a major prevailing problem”.
![Page 6: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/6.jpg)
History Historical trends in test automation frameworks:
1993 – 20011st Gen – Modularity
Driven
2001 – 20052nd Gen – Data
Driven
2005 – 20083rd Gen – Keyword Driven
2008 – 20114th Gen – Hybrid
Keyword Data Driven
• 1993 – 1999: WinRunner x.x• 1999 – 2001: WinRunner 6.x• 2001 – 2004: WinRunner 7.x,
RobotJ 1.0, XDE Tester 1.0• 2004 – 2005: QTP 7.x, RFT 6.x• 2004 : Selenium 1
• 2005 – 2006: QTP 8.x, RFT 7.x• 2006 – 2008: QTP 9.x, RFT 8.x• 2007: Selenium 2• 2008 – 2010: QTP 10.x, RFT 8.1• 2010 – 2011: QTP 11, RFT 8.2
![Page 7: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/7.jpg)
Myths and Truths1. Test Automation is simple, that every tester
can do it
Promoted by the Sales people by simply saying: Record the script Enhance the script by adding functions and data
driving Run the scripts Report results
Under this influence QA manager can proudly say “All our testers are doing test automation”.
![Page 8: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/8.jpg)
Myths and Truths But in Reality: TA is a software development task
It should be designed, developed and tested Need to have some kind of a programming
background to implement test automation. TA is not as complex as C++/C#/Java development.
TA components are assets that should be treated like application source code
Don’t fall into tool vendor sales pitch …remember Record & Playback is not real test automation
![Page 9: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/9.jpg)
Myths and Truths2. Commercial TA tools are expensive
Under the influence of this myth some companies, especially the small ones: Try to develop their own test automation tools Use scripting languages like Perl and Ruby Use shareware test tools Do not consider test automation at all
![Page 10: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/10.jpg)
Myths and Truths But in Reality: Commercial TA tools are not
that expensive
Per seat license for the most expensive automation tool is $8K, which can be used for 5 years.
Maintenance/Support fees are 20% of tool cost or $1,600 per year
The cost of this tool is $8K/5+$1,600 = $3,200 per year The automation developer cost with overhead is $100K
per year The cost of this tool is just 3% of the person who uses
it, but productivity gain can be very significant
![Page 11: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/11.jpg)
Myths and Truths Truth: 92% FAIL to meet target ROI
2004 2010 (Estimated)
Industry: Test Automation (Net Worth)
$1 Billion $ 6.3 Billion
Automation Projects (Failure Cost)
$ 0.6 Billion $ 3.8 Billion
40%
60%
Automation Projects
WorkingFailure
Source 1: http://www.nytimes.com/2006/07/26/technology/26hewlett.html Source 2: http://www.slideshare.net/Jonathon_Wright/hybrid-keyword-data-driven-automation-frameworks-jonathon-wright
32%
8%
ROI
Learning from past experience
![Page 12: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/12.jpg)
Why do Automation Projects typically Fail? IDT study(www.idtus.com)
Lack of Time37%
Lack of Expertise20%
Lack Of Budget
17%
Tool In-compatibil-
ity11%
Misc15%
![Page 13: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/13.jpg)
Why do Automation Projects typically Fail? Lack of defined automation
methodology Automation is not treated as a
legitimate project with the necessary planning / resources
Test Automation is typically performed at the end of the SDLC
After the initial success the automation scripts are not maintained for future builds
![Page 14: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/14.jpg)
Why do Automation Projects typically Fail? Testers are typically untrained in test tools
and programming techniques No modularization (reusable functions) in
automation scripts Automated tests cases are usually designed
based on front end functionality (black box testing)
![Page 15: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/15.jpg)
What is Software Test Automation? Test Automation is the use of software to
execute tests without Human intervention It refers to the activities and efforts that
intend to automate engineering tasks and operations in a software test process using well-defined strategies and systematic solutions.
Not like Rube Goldberg cartoons
![Page 16: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/16.jpg)
Why TA is needed? Objectives:
To free engineers from tedious and redundant manual testing operations
To speed up a software testing process, and to reduce software testing cost and time during a software life cycle
To increase the quality and effectiveness of a software test process by achieving pre-defined adequate test criteria in a limited schedule
![Page 17: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/17.jpg)
Key to Success
To reduce manual testing activities and redundant test operations using a systematic solution to achieve a better
testing coverage.
![Page 18: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/18.jpg)
Manual vs. Automated Testing Manual Testing:
Testing time is consuming and tedious Inefficient in today’s shorter SDLC Delay the ability to thoroughly test an application Critical bugs escape undetected What happens when multiple platforms involved
Automated Testing: Higher efficiency that accelerate the testing cycle and
promote software quality Optimizes software quality and testing efficiency by delivering
Reusability Predictability and Consistency Productivity
Enables accurate assessment of quality level
![Page 19: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/19.jpg)
Pros and Cons
Pros:SpeedReusabilityAccuracyRun AnytimeEfficiency
![Page 20: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/20.jpg)
Pros and Cons
Cons:Significant InvestmentMaintenanceNot as RobustError DetectionCannot Think
![Page 21: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/21.jpg)
What to Automate? Regression Tests: Stabilized tests that verify stabilized
functionality
Tests rerun often: Tests that are executed regularly vs. rarely
Tests that will not expire shortly: Most tests have a finite lifetime during which its automated script must recoup the additional cost required for its automation
Tedious/Boring tests:
tests with many calculations and number verifications
repetitive tests performing the same operations over and over
tests requiring many performance measurements
Just plain boring tests
Reliably repeatable
![Page 22: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/22.jpg)
What NOT to Automate? Unstable functionality: Not reliably repeatable
Rarely executed tests: poor Return-On-Investment
Tests that will soon expire: poor Return-On-Investment
Requiring in-depth business analysis:
some tests require so much business specific knowledge that it becomes prohibitive time wise to include every verification required to make its automated script robust enough to be effective
exceedingly complex tests are sometimes not possible to automate because computers cannot think
![Page 23: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/23.jpg)
![Page 24: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/24.jpg)
Contents BEP and ROI
BEP BEP - Example ROI PayBack Period Classic ROI Real ROI Benefits of ROI ROI Calculator
![Page 25: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/25.jpg)
BEP
Break-Even Point (BEP) is the point at which cost or expenses and revenue are equal: there is no net loss or gain
![Page 26: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/26.jpg)
BEP - Example
Preparation (V)(in Mins)
Execution (D)(in Mins) ROI
using Manu
al
Resources (R ) for (n) Automated TestsRn = Aa / Am = (Va + n*Da) / (Vm +
n*Dm)
TestManual Automated Manual Automated 1 5 10 20
Scenario 1 30 60 11 1.1 33% 149% 77% 51% 33%
Scenario 2 30 60 11 1.1 33% 149% 77% 51% 33%
Scenario 3 30 60 9 0.9 36% 156% 86% 58% 37%
Scenario 4 30 60 10 1 34% 153% 81% 54% 35%
Scenario 5 30 60 10 1 34% 153% 81% 54% 35%
Scenario 6 30 60 10 1 34% 153% 81% 54% 35%
Scenario 7 30 60 15 1.5 27% 137% 64% 42% 27%
Scenario 8 30 60 30 3 5% 105% 42% 27% 19%
Scenario 9 30 60 22 2.2 16% 120% 51% 33% 22%
Scenario 10 30 60 12 1.2 31% 146% 73% 48% 31%
Total 300 600 140 14 28% 142% 71% 47% 31%
![Page 27: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/27.jpg)
ROI Return on Investment ROI = BENEFIT/COST ROI = (total benefit – total cost) / (total cost) ROI = (cost of manual – cost of automation) /
cost of automation Where,
Automation Cost = Price Of HW + Price of SW + Development Cost + Maintenance Cost + Execution Cost
Manual Testing Cost = Development Cost + Maintenance Cost + Execution Cost
Looks right, Doesn’t it?
![Page 28: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/28.jpg)
![Page 29: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/29.jpg)
Classic ROI Problems with classic ROI calculation:
You can’t compare Automated Testing and Manual Testing. They are not the same and they provide different information about the AUT.
You can’t compare cost of multiple execution of automated tests vs. manual tests. You would never dream of executing that many test cases manually
So then…what is real ROI?
![Page 30: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/30.jpg)
Real ROI ROI value is not the value of Automation vs.
Cost of executing these tests manually Automation ROI value is the benefit of this
type of testing, and it can be: Reducing Time to Market Increased Test Efficiency (Productivity) Increased Test Effectiveness
![Page 31: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/31.jpg)
Benefits of ROI Reduced Time to Market
Products delivered quickly Makes people available to work on other projects Higher margins, if no competitive products in
market Productivity and Effectiveness
More testing gets done faster, increasing the odds of finding defects
Defects found early have better chances of being fixed
Manual Testers can concentrate on clever ways to finding defects, instead of typing test inputs and verify output.
![Page 32: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/32.jpg)
Benefits of ROI About 7% of bug fixes create new bugs,
sometimes in already tested parts of the system. With automation you can rerun tests for those modules. This almost never happens when testing is done manually.
![Page 33: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/33.jpg)
ROI Calculator ROI Calculator:
Source 1: http://www.aspiresys.com/testautomationroi/index.php
Source 2: http://www.elbrus.com/services/test_automation_roi_calc/
![Page 34: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/34.jpg)
ROI Calculator
![Page 35: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/35.jpg)
ROI Calculator
![Page 36: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/36.jpg)
Contents Tools
Choosing the right tool – The Strategic Approach
Choosing the right tool – The Strategic Steps Tools Classification Short List Tools – Functional Short List Tools – Performance Evaluate Vendors Functional Test Tools - Analysis Feasibility Analysis
![Page 37: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/37.jpg)
Tools
There is no single best testing tool; rather, different tools are more or less appropriate in different environments.
![Page 38: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/38.jpg)
Tools Over 300 Test Tools are available (
http://www.softwareqatest.com) Load/Performance tools – 54 Web Functional/Regression – 60 Java Test tools - 48 Other Web tools – 76
Which tool is right for you?
![Page 39: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/39.jpg)
Choosing the right tool – The Strategic Approach• Is there an organizational methodology for
test automation?• Which applications/processes?• What is the impact to current project
schedules?• What is the effort in maintaining
automated tests?• What are the costs?• What about tools integration?• What about Continuous Integration?
![Page 40: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/40.jpg)
Choosing the right tool – The Strategic Steps• Step 1: Define and Refine Requirements• Step 2: Communicate the Impact• Step 3: Develop Evaluation Methodology • Step 4: Select, Procure, and Implement
![Page 41: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/41.jpg)
Step 1: Define and refine requirement Create a list of organizational requirements
What problems do you want the tool to solve? What capabilities will the tool need to be
effective in your environment? Other lifecycle tools?
What constraints, budgetary or otherwise? Identify compatibility issues
What operating systems does your application support?
What is the development environment? Does your application integrate with third-party
software? Does the application use custom controls?
![Page 42: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/42.jpg)
Step 1: Define and refine requirement Identify tool audience
Who will use the tool on a day-to-day basis? What is the level and mix of user skill levels? Is your organization willing to invest in training?
Define technical or business requirements Does your organization have additional
requirements? Software standards Technical standards Procurement rules Preferred vendor rules
![Page 43: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/43.jpg)
Step 1: Define and refine requirement Identify budget constraints
How much can we afford? How much is this worth?
New requirements may surface based on research
Did not know about Forgot to include
![Page 44: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/44.jpg)
Step 2: Communicate the Impact Automated testing is part of the larger
strategic application development endeavor Communicate the effects of implementing a tool
Chance to discuss and mitigate concerns
How tool may change job description Commitment to training Implementation strategy
Discussion may imply additional requirements Business, functional, technical, or operational
![Page 45: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/45.jpg)
Step 3: Develop the evaluation methodology How will tools be compared? Are there specific features that may
differentiate one tool from another? Are there specific things that can eliminate a
tool from consideration? Preferred vendor list can reduce evaluation
scope Demos and evaluations are time-consuming Identify a representative set of activities to
accomplish with the tool during the evaluation
![Page 46: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/46.jpg)
Step 4: Select, procure, and implement Make an informed selection Follow organization’s procurement process Develop the implementation plan
What When Why Who How
![Page 47: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/47.jpg)
Step 4: Select, procure, and implement Develop an implementation plan
Enterprise applications requiring multiple releases Applications that must produce a consistent set of
results using stable data These characteristics fully leverage reusability and
predictability benefits of automated testing
Take implementation one step at a time Take time for training Keep focus on staff issues and reactions
![Page 48: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/48.jpg)
Step 4: Select, procure, and implement Develop a test plan
Describes scope, approach, resources and schedule for all automated and manual activities Rule of Thumb: (Test scripts) 40% manual - 60%
automated
Create and deploy your automated tests Be selective with the automation of test scripts
Verify the most critical functionality Are the most likely to expose defects Are expensive or impossible to perform manually
Use the first automated suites you build for Smoke testing Regression testing
![Page 49: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/49.jpg)
Decision?How to calculate the cost of functional test
automation
If a test script will be run every week for the next 2 years, automate the test if thecost of automation is less than the cost of manually executing the test 104 times.
Cost of automation Cost of manually executing the test as manytimes as the automated test will be executed
Cost of test automation
Cost of tool(s)
Labor costs of script creation
Labor costs ofscript maintenance
Automate if
![Page 50: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/50.jpg)
Tools Classification
Test Tool Types Basic Descriptions of Different Types of Test Tools
Test InformationManagement
Systematic solutions and tools support test engineers and quality assurance people tocreate, update, and maintain diverse test information, including test cases, test scripts,test data, test results, and discovered problems.
Test Execution andControl
Systematic solutions and tools help engineer set up and run tests, and collect andvalidate test results.
Test Generation Systematic solutions and tools generate program tests in an automatic way.
Test CoverageAnalysis
Systematic solutions and tools analyze the test coverage during a test process basedon selected test criteria.
Performance Testingand Measurement
Systematic solutions and tools support program performance testing and performancemeasurement.
Software Simulators Programs are developed to simulate the functions and behaviors of external systems,or dependent subsystems/components for a under test program.
Regression Testing Test tools support the automation performance of regression testing and activities,including test recording and re-playing.
![Page 51: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/51.jpg)
Tools Classification
Types of Test Tools Test Tool Vendors Test Tools
Problem Management Tools Rational Inc. ClearQuest, ClearDDTS
Microsoft Corp. PVCS Tracker
Imbus AG Imbus Fehlerdatenbank
Test Information Management Tools
Rautional Inc. TestManager
Mercury Interactive TestDirectory
Test Suite Management Tools Evalid TestSuiter
Rational Inc. TestFactory
SUN JavaTest, JavaHarness White-Box Test Tools McCabe & Associates McCabe IQ2
Junit
IBM IBM COBOL Unit TesterIBM ATC- Coverage Assistant- Source Audit Assistant- Distillation Assistant- Unit Test Assistant
![Page 52: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/52.jpg)
Tools ClassificationTest Execution Tools OC Systems Aprob
Softbridge ATF/TestWright
AutoTester AutoTester
Rational Inc. Visual Test, Rational Functional Tester
SQA Robot
Mercury Interactive WinRunner, Quick Test Prof
Sterling Software Vision TestPro
Compuware QARun
Seque Software SilkTest
RSW Software Inc. e-Test
Cyrano Gmbh Cyrano Robot Code Coverage Analysis Tools Case Consult Corp. Analyzer, Analyzer Java
OC Systems Aprob
IPL Software Product Group Cantata/Cantata++
ATTOL Testware SA Coverage
Compuware NuMega TruCoverage
Software Research TestWorks Coverage
Rational Inc PureCoverage
SUN JavaScope
ParaSoft TCA
Software Automation Inc Panorama
![Page 53: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/53.jpg)
Tools Classification
Load Test and Performance Tools Rational Inc. Rational Performance Tester
InterNetwork AG sma@rtTest
Compuware QA-Load
Mercury Interactive LoadRunner
RSW Software Inc. e- Load
SUN JavaLoad
Seque Software SilkPerformer
Client/Server Solutions, Inc. Benchmark Factory
Regression Testing Tools IBM Regression Testing Tool(ARTT)
Distillation Assistant
GUI Record/Replay Software Research eValid
Mercury Interactive Xrunner
Astra Astra QuickTest
AutoTester AutoTester, AutoTester One
![Page 54: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/54.jpg)
Short List Tools - FunctionalToolVendor Test Suite - Companion Tools
TestPartnerCompuware QACenter Enterprise Edition+
QuickTest Professional
Mercury Quality Center
e-TesterEmpirix e-TEST suite
Rational Functional Tester
IBM Rational Suite
WebFTRadView TestView Suite
SilkTestBorland(Segue)
SilkCentral Test Manager
QA Wizard ProSeapine TestTrack Pro
![Page 55: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/55.jpg)
Short List Tools - Performance
ToolVendor Test Suite - Companion Tools
QALoadCompuware QACenter Enterprise Edition+
LoadRunnerMercury Quality Center
e-LoadEmpirix e-TEST suite
Rational Performance Tester
IBM Rational Suite
WebLOADRadView TestView Suite
SilkPerformerBorland(Segue)
SilkCentral Test Manager
ForecastFacilita ForecastWeb, ForecastNet, ForecastDB
![Page 56: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/56.jpg)
Evaluate VendorRisky bets Contenders
Currentofferings
StrategyWeak Strong
StrongLeaders
Strong performers
IBM
Empirix
Seapine
Compuware
Mercury
RadView
Segue
![Page 57: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/57.jpg)
Functional Test Tools - Analysis
Tool Pros ConsIBM/Rational Functional Tester (RFT)
•Built as Eclipse Plug-In with full IDE and Java support•Supports Web 2.0, Java or .NET applications•Full GUI Object Map repository
•Insufficient browser support•Licensed product
HP/Mercury Quick Test Pro (QTP)
•Supports Web 2.0, Java or .NET applications•Full GUI Object Map repository•Seamless integration with QualityCenter
•VisualBasic scripting is limited•No IDE (may change in new release)•Licensed Product
Selenium RC & IDE
•Good browser support•Good language support (Java, Ruby,C# )•Can be easily extended as JUnit suite•Open-source
•No GUI Object repository•Only web application support
![Page 58: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/58.jpg)
Feasibility Analysis FA Matrix Available Operational Feasibility Technical Feasibility Economic Feasibility Schedule Feasibility
![Page 59: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/59.jpg)
FA Matrix
![Page 60: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/60.jpg)
FA Matrix
![Page 61: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/61.jpg)
Contents Automation Framework
Where to start? What is AF? Benefits of AF How does it work? Architecture Framework Approaches
Record and Playback Script Based Approach Keyword Driven Approach Data Driven Approach Hybrid Keyword and Data Driven Approach
![Page 62: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/62.jpg)
Where to Start?
First find out Then you can work out
What needs to be tested?
What needs be automated?
What can be tested? What can be automated?
What could be tested? What could be automated?
“Start SMALLthink BIG”
Quick winsshould beavoided
NEVER expect toautomate 100%
“Under promise,Over deliver?”
Keep it simple,wherever possible
Focus on keycritical businessprocesses
![Page 63: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/63.jpg)
What is Automation Framework? Framework – independent of application or
environment under test A Test Automation Framework is a set of
assumptions, concepts and tools that provide support for Automated Software Testing.
A reusable set of libraries or classes for a software system (or subsystem).
A correctly implemented Test Automation Framework can further improve ROI by reducing the development and maintenance costs.
![Page 64: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/64.jpg)
Benefits of Framework Ease of Use – easy to learn and easy to use Time – faster than capture/replay and scripting approach Maintainability – significantly reduces the test maintenance
effort Reusability – due to modularity of test cases and library
functions Manageability - effective test design, execution, and traceability Accessibility – to design, develop & modify tests whilst
executing Availability – scheduled execution can run unattended on a 24/7
basis Reliability – due to advanced error handling and scenario
recovery Flexibility – framework independent of AUT or environment Measurability – customizable reporting of test results ensure
quality Increases the ROI (Return on Investment)
![Page 65: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/65.jpg)
How does it work? Different Implementations
One Example of Keyword Driven Framework could be: Spreadsheets, Spreadsheets, Spreadsheets Test Objects Keywords and Methods = Toast! Parameters Description or Call the 911?
![Page 66: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/66.jpg)
Architecture
![Page 67: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/67.jpg)
Framework Approaches Record and Playback Script Based Approach Keyword Driven Approach Data Driven Approach Hybrid Keyword and Data Driven Approach
![Page 68: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/68.jpg)
Manual Testing – Looking back
+ easy & cheap to start
+ flexible testing
- expensive every execution
- no auto regression testing
- less coverage measurement
![Page 69: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/69.jpg)
Record and Playback
+ flexible testing
- expensive first execution
+ auto regression testing
- fragile tests break easily
- less coverage measurement
![Page 70: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/70.jpg)
Script Based Approach+/- test impl. = programming
+ automatic execution
+ auto regression testing
- fragile tests break easily?
(depends on abstraction)
- less coverage measurement
![Page 71: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/71.jpg)
Data Driven Approach Automation is data-centric User defines just data sets to drive tests with Will have an external data source (DB tables,
Excel spreadsheets, XML for data sets) Flow control (navigation) is normally done by
the test script not by the data sourceEx: data set exercises creation of new sales
accounts functionality; stored in a DB table account_dataCompanyName PrimarySalesPerson Street Zip City State
Genesis Inc. Phil Collins 5775 Main st 30075 Atlanta GA
RollingStones Inc. Mick Jagger Jr. 2332 Washington st 02111 Boston MA
![Page 72: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/72.jpg)
Keyword/Action Driven Approach+ abstract tests
+ automatic execution
+ auto regression testing
- robust tests
- less coverage measurement
![Page 73: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/73.jpg)
Keyword/Action Driven Approach Automation is action-centric De-compose your test cases/modules into granular re-
usable keywords The idea is for non-coders to be able to create
automated test cases with action keywords User defines flow control of the test via action
keywords
Example: Test Case “Verify Checking Account Balance”1. Enter Username and Password and Click submit button step
1 is action Login2. Enter “Phil Collins” as a Sales Person and Click Submit button3. Verify the Sales Person was successfully created and Logout
So you may want to choose the following re-usable action keywords:
EnterText, Click, Login, VerifyExists
![Page 74: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/74.jpg)
Benefits of Keyword Driven Approach This Framework addresses the most
common problem with test automation: Automation Engineers do not have domain knowledge and the End Users (Subject Matter Experts/Test Engineers) usually do not have automation expertise.
When properly implemented and maintained, it presents a superior ROI because each business event is designed, automated and maintained as a discrete entity.
Keywords can then be used to design test cases, but the design and automation overhead for the keyword has already been paid.
![Page 75: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/75.jpg)
Benefits of Keyword Driven Approach Reduced the cost and time spent maintaining and
updating tests The modular structure of keyword-driven testing
means that new tests can easily be created from pre-existing modules
The test team is capable of entirely automating tests, even without programming knowledge
Can be easily modified to use with different test tool Reusability across different projects
Classic Example:Object Action Data
Textfield (username)
Enter Text <username>
![Page 76: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/76.jpg)
Keyword/Action Driven Approach May have an external data source (DB tables, Excel
spreadsheets, XML for data sets) with action keywords
Step
Description Page Action
Module
Type Object Expected
1 Login Home LoginUserLogin N/A
.id:=LoginSubmit
.text:=Login Successful
2
Enter New Sales Person data
CreateSalesPerson EnterText Field
.text:=SalesPerName
.value:=Phil Collins
3Click Submit
CreateSalesPerson Click
Button
.id:=SubmitSalesPer
.url:=.*createdSalesPerStatus.html
4
Verify Sales Person Creation Successful
CreateSalesPersonStatus VerifyExi
sts DIV.id:=CreationStatus
.value:=User Created Successfully
![Page 77: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/77.jpg)
Hybrid Keyword and Data Driven Approach Combines the best of both worlds
User defines data sets to drive tests with User also defines flow control of the test via
action keywords May have an external data source (DB tables,
Excel spreadsheets, XML for data sets) with action keywords in addition to generic and test case specific data sets
![Page 78: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/78.jpg)
Architecture
![Page 79: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/79.jpg)
Open2Test Automation Framework
![Page 80: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/80.jpg)
Open2Test Automation Framework
![Page 81: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/81.jpg)
Open2Test Automation Framework
![Page 82: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/82.jpg)
Model Based Approach+ abstract tests
+ automatic execution
+ auto regression testing
+ auto design of tests
+ systematic coverage
+ measure coverage of model and requirements
- modelling overhead
Emerging Approach
![Page 83: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/83.jpg)
References http://www.slideshare.net/Jonathon_Wright/hy
brid-keyword-data-driven-automation-frameworks-jonathon-wright?src=related_normal&rel=805408
http://www.ibm.com/developerworks/rational/library/591.html
http://www.keane.com/resources%2Fpdf%2FWhitePapers%2FWP_ROIforTestAutomation.pdf
![Page 84: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/84.jpg)
Summarizing… Overview of STA
History Myths and Truths Why Automation Projects Fail? What is TA? Why TA is needed? Manual vs. Automated Pros & Cons What to Automate? What Not to Automate?
BEP and ROI BEP BEP - Example ROI Classic ROI Real ROI Benefits of ROI ROI Calculator
Tools Choosing the right tool – The Strategic
Approach Choosing the right tool – The Strategic
Steps Tools Classification Short List Tools – Functional Short List Tools – Performance Evaluate Vendors Functional Test Tools - Analysis Feasibility Analysis
Automation Framework Where to start? What is AF? Benefits of AF How does it work? Architecture Framework Approaches
Record and Playback Script Based Approach Keyword Driven Approach Data Driven Approach Hybrid Keyword and Data Driven Approach
![Page 85: Software test automation_overview](https://reader035.fdocuments.net/reader035/viewer/2022062303/55549c6ab4c905fd608b477c/html5/thumbnails/85.jpg)
THANK YOU
Rohan Bhattarai