The Future of Test AutomationThe Future of Test Automation

Worksoft, Inc.© 2003 All Rights Reserved

Agenda

Business Case for Test Automation– Why automate?

Test Automation Challenges– Common mistakes

Software Development Trends– Risk is increasing

Test Automation Trends– Technology is evolving

Future of Test Automation– Quality maturity

Business Case forTest Automation

Why Automate?

Save timeLeverage resourcesReduce costsIncrease coverage

Time

Repeatable tests– Multiple builds– Multiple versions

Unattended execution– Time to design more tests– Time to do manual testing

Concurrent testing– Write tests during requirements– Develop tests during coding

Resources

Reproducible tests– Audit trail of test results– Consistency from one tester to another

Higher volumes– Acquire test data from production– Generate data programmatically

Reduced rework– Catch errors early– Fewer emergencies, patches

Costs

Reduce failures in production– Loss of revenues, market share– Loss of customers, confidence– Loss of user productivity

Reduce support, rework costs– Costs to recover from errors– Costs to remedy defects– Costs to retest, redeploy

Coverage

Functionality increases over time– Inventory of existing features is cumulative– New features are added– Fixes must be verified– Ad hoc tests are needed

Do the math– Cycle times are shorter– Developers outnumber testers– A 10% software change requires 100% testing– A single release will have multiple builds

Features

Tests

Time#

Risk

Test Automation Challenges

Common Mistakes

You don’t have timeYou don’t have resourcesYou don’t understand costsYou can’t measure coverage

Time is Limited

Shorter cycle times– Schedules are market-driven, not effort-based

– Original schedule is always compressed

– Poor quality means longer cycles

Test automation takes time– It’s a development project

– 5X to 10X manual effort

– Ongoing maintenance effort

Resources are Limited

Recording tests takes manual effort– Tests have to be performed manually– You can’t start until software is ready

Scripting takes skill– Tools are programming languages– Automation is a development project

Implementation takes resources– Design, document tests– Develop framework, scripts– Execute, evaluate results

Costs Aren’t Fully Apparent

Cost of acquisition– Educate, evaluate, negotiate, license

Cost to implement– Planning, training, design, development,

documentation

– Requires development resources

Cost of ownership– Continuous maintenance, support

– Change, version control

– Training for new personnel

Coverage is Difficult to Measure

Requirements are missing, obsolete– Test cases are ad hoc– Test knowledge is organic

Regression tests are undefined– Existing functionality not captured– Past defects are forgotten

New features are informal– Poor or missing specifications– Casual or inconsistent hand-off

Software Development Trends

Risks are Increasing

Increased exposureDecreased cycle timesGreater functionalityMore complexity

Increased Exposure

Business rules exposed to users– Rules engines enable analyst involvement– More changes made faster

Applications exposed to customers– Web access becomes customer interface– Direct impact on credibility

Applications exposed to suppliers– Web services open applications– Direct impact on production

Decreased Cycle Times

Market-driven schedules– Software is competitive weapon– Customer, competition drive demands– Supply chain integration drives dates

Accelerated development techniques– Agile, extreme programming practices– Short interval, incremental deliveries– Cycles times of weeks or months

Greater Functionality

Component-based development– Incorporate new functionality quickly– Less development, more testing

Integrated applications– Massive ERP, CRM systems– Easy to configure, difficult to test

Integrated enterprises– Supply chain integration– Standards for development, none for test

Test Automation Trends

Technology is Evolving

Record/ReplayRecord/Replay/ScriptingData-Driven/Keyword Driven

FrameworkClass/Action Framework

Capture/Replay

Test cases executed manually and recorded– No leverage– No concurrent test development

Scripts are unreliable– Sensitive to timing, changes– No logic for error handling

Scripts are not maintainable– Unstructured, undocumented– Hard coded data means huge volumes

Example Script

# Add Accountset_window ("Account Manager", 15);edit_set ("txtName", “Worksoft");obj_type ("txtName","<kTab>");edit_set ("txtAddress", “123 Main Street");obj_type ("txtAddress","<kTab>");edit_set ("txtCity", “Dallaslist_select_item ("cbState", "Texas"); # Item Number 42;edit_set ("Edit", "Texas");edit_set ("txtZip", “99999");obj_type ("txtZip","<kTab>");button_press ("Save_2");

Record/Replay/Scripting

Test cases executed manually, recorded then modified– No concurrent test development

– Requires programming skills

Scripts are complex– Writing programs to test programs

– Code added for synchronization, logic, error handling

– Code added for external data

Scripts are difficult to maintain– Script code can exceed application code

– Turnover means obsolescence

Example Script# Add Account

set_window ("Account Manager", 15);if ( obj_exists “txtName != E_OK ){

gsStepStatus = "edit_Input: Edit object does not exists";return;}status = edit_set ( sAccountName );

}if ( status != E_OK ){

gsStepStatus = "edit_Input: Value could not be input";return;

}type("<kTab>");

Data-Driven/Keyword Driven Framework

Keywords recorded or generated, then modified– Script language specific– Requires programming skills

Scripts are complex, high volume– Complete framework must be developed– May be hundreds of action/keyword variations

Scripts are difficult to maintain– Added or modified for new functionality– Multiple changes for application UI changes

Keyword Framework Structure

Proprietary Tool

Business Processes

Execution ScriptsData Files

Configuration Managed Assets

Keyword Functions

Calls to BPs

Object Map

Example Test Case

Begin testcase Cust-AUD

Name Address City State Zip

Add Account Worksoft 123 Main Dallas Texas 77777

Update Account Acme AC 456 Elm Phoenix Arizona 88888

Delete Account Bay Brick 789 Shore San Jose California 99999

Class/Action Framework Document then execute

– Pre-written libraries, customer-extensible– Proven, supported application

Point and click development– Enforced structure, conventions– Add, manage data easily

Powerful framework– Synchronization, error handling, recovery– Standard and shared processes

Relational database– Automated impact analysis, maintenance– Multi-user, centralized maintenance, administration– All test assets can be housed in a database—no flat files

Open architecture– Single interface across multiple platforms, tools

Class/Action Framework Structure

Business Processes

Data (in DB)

Configuration Managed Assets

Proprietary Tool

Action FunctionsObject

Map

Business Processes and data are decoupled from tool

Application Map

Define Process

Add Application

Add Window

Select Object

Select Action

Input Value

Literal data

Variable data

Generic Functions

public function edit_Input ( sObjectId, sObjectName, sWindowId, sWindowName ){

if ( obj_exists ( sObjectId ) != E_OK ) {gsStepStatus = "edit_Input: Edit object does not exists";return;

} if ( GetStringArgument ( "Value", gsExpectedValue, sDefaultValue ) == FAILED )

{gsStepStatus = "edit_Input: Action parameter (Value) could not be retrieved";return;

}status = edit_set ( sObjectId, gsExpectedValue );

if ( status != E_OK ){

gsStepStatus = "edit_Input: Value could not be input";return;

}type("<kTab>");gbActionStatus = PASSED;return;

}

Specify Result on Pass, Fail

Specify Log Result

Add Data

Automated Documentation

Process / Window Step Description*Account Master

Press New CommandButtonInput Worksoft into Account Name TextBoxInput 123 Main into Account Address TextBoxInput Dallas into Account City TextBoxSelect Item Texas from Account State ListBoxPress Save CommandButton

*MessageBoxVerify Text is Equal To “Account Added”Press OK CommandButton

Automated Impact Analysis

Automated Maintenance

Extensive Reporting

Quality Maturity

Proprietary languages are being retired– Shift to native languages

Testability is becoming accepted– Shift to cooperative development

Automation is becoming automated– Tools are maturing

Increased management awareness– Risks are higher

Shift to Native Languages

Proprietary languages too costly– Skilled resources are scarce– 10X development costs– Deployment license costs– Not all tools support all platforms

Major vendors in transition– Mercury: WinRunner to QuickTest Pro/VBScript– Compuware: QA Run to Test Partner/VBA– Rational: Robot VB to Java

Shift to Cooperative Development

Development practices improving– Coding standards, conventions– Object properties, methods exposed– Unique, persistent, meaningful names

Development support enhanced– Code instrumentation more likely– Standards compliance more rigorous– Test APIs more common

Tools are Maturing

Next generation automation products– Certify– Unified TestPro, ABT, TestFrame

Automated unit testing – JUnit, NUnit– .TEST– AUTT

Testing designed into architectures– Test standards for Web services

Risks are Higher

Sarbanes Oxley raises stakes– CXO-level responsibility for financial accuracy– Duty imposed for internal controls– Potential civil, criminal liability

Customer, exterprise integration raises exposure– Defects are visible to outside world– Errors can impact entire supply chain

Build/Buy the Right Framework

Less development, more testing– Introduce development to testability– Focus test engineers on changes and exceptions– Empower business analysts to automate

Reduce cost of implementation, ownership– Less training, faster deployment– Less maintenance, support– Lower tool license costs

Introduce enterprise standards– Enable seamless end to end execution– Provide portability across tools, platforms– Leverage resources across applications