Test Tool Evaluations - UML Testing Tool Evaluation Matrix.pdf · 248 Appendix B Test Tool...

a p p e n d i x b

Test Tool Evaluations

Anyone who has contemplated the implementation of an automated test tool has

quickly realized the wide variety of options on the market in terms of both the kinds

of test tools being offered and the number of vendors. The best tool for any partic-

ular situation depends on the system engineering environment that applies and the

testing methodology that will be used, which in turn will dictate how automation

will be invoked to support the process.

This appendix evaluates major tool vendors on their test tool characteristics,

test execution capability, tool integration capability, test reporting capability, per-

formance testing and analysis, and vendor qualification. The following tool vendors

evaluated are Compuware, Empirix/RSW, Mercury, Rational, and Segue.

243

s

R

L

D

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 243

S

R

L

Appendix B Test Tool Evaluations244

Intuitive and easy to use for users newto automated testing tools

Intuitive and easy to use for users newto this particular tool but who haveused others

Easy to install; tool may not be used ifdifficult to install

Tasks can be accomplished quickly,assuming basic user proficiency

Easy to maintain automated tests witha central repository whereby users canseparate GUI object definitions from thescript

Can vary how designs and documentsare viewed (zooming, multipagediagrams easily supported, multipleconcurrent views up at the same time);basic windowing

QACenter—YesPerformance Edition—Yes





NA

eTest Suite—YesUses Visual Scripts whichrequire no programming.

eTest Suite—YesMaintainable VisualScripts, fully extensibleprogramming

eTest Suite—YesNo proxy servers or specialconfigurations

eTest Suite—Yes

eTest Suite—YesUse of shared workspace,VBA code shared betweenscripts

eTest Suite—YesView test results/datain MDI windows, andswitch between

Table B.1 Test Tool Characteristics:Capture/Playback and Script Creation

Criterion/Feature Compuware Empirix/RSW

Ease of Use

Tool Customization

Fully customizable toolbars to reflectany commonly used tool capabilities

Tool customizable: fields added,deleted

Performance Edition—Yes

NA

eTest Suite—Notcustomizable, butcommon tool capabilitiesin the toolbar

NA

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM 44

s

R

L

245B.1 Test Tool Characteristics

Astra QuickTest—Yes

Astra QuickTest—YesWinRunner—Yes





Rational Suite TestStudio, RationalRobot—YesGUI interface, popup icon bars, tool tipsand on-line help.

Rational Suite TestStudio, RationalRobot—YesSQA Robot used for both GUI functional& performance tests.

Rational Suite TestStudio, RationalRobot—YesAn administrator should be assigned tohelp set up test asset datastore &license server.

Rational Suite TestStudio, RationalRobot—Yes

Rational Suite TestStudio, RationalRobot—YesScripts are maintained in a central datastore.

Rational Suite TestStudio, RationalRobot—YesDockable windows, toolbars and otherattributes allow interfacecustomization.

SilkTest—Yes

SilkTest—Yes

SilkTest—Yes

SilkTest—Yes

SilkTest—Set up so that bydefault, object declarations are inseparate include file soscript can have minimum codeand intuitive steps inside the testcases.

SilkTest—Uses format similar toWord’s outlining. Can zoom in andout of scoping by clicking onpluses and minuses. Can viewmany documents.

B.1 Test Tool Vendors: Compuware, Empirix and Mercury

Mercury Rational Segue

Ease of Use

(continued )

Tool Customization


Astra QuickTest—YesWinRunner—YesTestDirector—Yes

Rational Suite TestStudio—YesToolbars, menu bars and icon bars arecustomizable.

Rational Suite TestStudio—YesThree custom fields used to elaborateon any test asset.

SilkTest—No

Silk Test—YesFeatures are customizable, but notthe GUI.

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 245

S

R

L


Fully customized editor with formatsand colors for better readability

Tool support for required test procedurenaming convention



eTest Suite—YesCode editor fullycustomizable

eTest Suite—Yes

Table B.1 Test Tool Characteristics: Capture/Playback and Script Creation (continued)


Tool Customization

Platform Support

Multiple platform support: UNIX,XWindows, Windows CE, Win3.1,Win95, Win98, NT, Win2000, WinME

Browser support: all versions ofNetscape, Internet Explorer

Cross-browser testing (record on IE,replay on NS, or vice versa) capability

Technology support for one or more ofHTML, DHTML, JavaScript, XML, Javaapplications and applets, MFC (C/C++),Visual Basic, PowerBuilder, Delphi,Terminal Emulator, Oracle, SAP,PeopleSoft, Siebel

QACenter—Supports variousWindows versions. PerformanceEdition supports Unix and variousWindows. Can also test serversserving content for mobile devices.


NA

QACenter—YesPerformance Edition—YesSupports Java apps and applets ina customer’s JVM. Supports HTML,DHTML, JavaScript, XML, SAP,Oracle, PeopleSoft, Siebel

Supports Unix andvarious Windows.

eTest Suite—Yes

eTest Suite—Yes

eTest Suite—YesSupports Java apps andapplets w/in MicrosoftJVM/Sun JVM. HTML,DHTML, JavaScript, XML,MFC, Visual Basic,Oracle, SAP, Delphi,Terminal Emulator,PowerBuilder, Oracle,PeopleSoft

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 246

s

R

L




Rational Suite TestStudio, RationalRobot—YesRobot editor is color-coded. Reservedwords highlighted—blue, comments—green.

Rational Suite TestStudio, RationalRobot—YesNames up to 40 characters.Dashes / Numeric okay.

SilkTest—Good editor

SilkTest—YesAlphanumeric with no space.



Tool Customization

(continued )

Platform Support

WinRunner—variousWindowsAstra QuickTest—supportsvarious Windows.Astra LoadTest—NTXRunner—HP, IBM, SunLoadRunner—NT, Unix

WinRunner—YesAstra QuickTest—Yes


WinRunner—Yes HTML, DHTML, JavaScript,VBScript, XML, Java applets,ActiveX, Visual Basic,PowerBuilder, Delphi,Terminal Emulator, Oracle,SAP, PeopleSoft, SiebelAstra QuickTest—HTML,DHTML, JavaScript, VBScript,XML, Java applets, ActiveX

Rational Suite TestStudio, RationalRobot—YesSupports UNIX and various Windows

Rational Suite TestStudio, RationalRobot—YesPresently not Netscape 6.0


Rational Suite TestStudio, RationalRobot—YesHTML, Dynamic HTML, JavaScript,VBScript, ActiveX, Java apps, Javaapplets

Supports UNIX and variousWindows

SilkTest—Yes

SilkTest—Yes

SilkTest: Yes—HTML, DHTML,JavaScript, XML, Java appsand applets, MFC, VisualBasic, Oracle, PowerBuilder,Delphi, SAP

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 247

S

R

L


Allows add-ins/extensions compatiblewith third-party controls

Does not involve additional cost foradd-ins and extensions

Test editor/debugger feature

Test scripting language flexible yetrobust; allows for modular scriptdevelopment

Scripting language not too complex

Scripting language allows for variabledeclaration and use and for parametersto be passed between functions

A test script compiler or an interpreterused

TestPartner—Yes

NA



TestPartner uses VBA for scriptinglanguage.Performance Edition uses C, acommon language.QARun has its own scriptinglanguage.

QACenter—YesPerformance Edition—YesLanguage also supports COMtesting

QARun and TestPartner use aninterpreter.Performance Edition—Scripts arecompiled for performance benefits.

eTest Suite—Yes

eTest Suite—Yes

eTest Suite—YesExtensive debugger

eTest Suite—YesVisual Scripts do notrequire a programminglanguage, but can beextended by using VB. Canuse script libraries.

VB—a common language

eTest Suite—Yes

eTest Suite—YesScripts are compiled forperformance benefits.



Test Language Features

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 248

s

R

L


WinRunner—Yes

WinRunner—Additional costsfor some environments


WinRunner—Yes uses TSL(Test Script Language)developed for testing.Astra QuickTest usesVBScript.

WinRunner TSL is easy touse, English-like interpreted,procedural language.Astra QuickTest usesVBScript also interpretedlanguage.


Mercury Interactive tools useinterpreted languages.

Rational Suite TestStudio, RationalRobot—YesSupported via followingutilities/functions: object mapping,object data test, definition, Java ProxyAPI, Custom Control API.

Rational Suite TestStudio, RationalRobot—No additional costs.

Rational Suite TestStudio, RationalRobot—YesColor-coded editor and debugger toinclude a compile option, debug runmode, go until cursor mode, step mode,breakpoints and a watch variableswindow.

Rational Suite TestStudio, RationalRobot—YesTwo scripting languages, both robustand extensible. Users can writefunctions, create libraries & callexternal API’s. Robot can also be utilizedusing Visual Basic and Java.

Rational Suite TestStudio, RationalRobot—SQABasic is a VB-likelanguage. VU is C-like language usedfor performance testing.


Rational Suite TestStudio, RationalRobot—YesSQABasic scripts compiled and theninterpreted at runtime. VU scripts arecompiled to an executable format.

SilkTest—YesAccomplished via object mapping,dll calls, or by adding an extensionvia our Extension Kit.

SilkTest—Yes

SilkTest—Yes

SilkTest—Yes

SilkTest—Not complex.Object Oriented languagestructure with ability to usefunctions. Very much like C++,not as rigid as Java.

SilkTest—YesStrong feature of SilkTest.

SilkTest—YesCompiler used for performance,manipuate the actual objects, andexercise actual AUT code.




(continued )

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 249

S

R

L


Interactive test debugging: allowsviewing variable values, steps throughthe code, integrates test procedures, orjumps to other external procedures

Allows recording at the widget level(object recognition level)

Allows for interfacing and testing ofexternal .dll and .exe files

Published APIs: Language InterfaceCapabilities

Tool is not intrusive: source code ofapplication needs to be expanded byinserting additional statements ordlls for the application to becompatible with the tool

Allows for data-driven testing

Allows for automatic data generation

Allows for adding timers for timingtransaction start and end

QARun—LimitedTestPartner—VBA full debuggerPerformance Edition—Limited

NA

TestPartner—Yes Performance Edition—Yes

TestPartner—Yes


QACenter—Yes Performance Edition—Yes


Performance Edition—YesQACenter—YesQALoad—Yes

eTest Suite—YesVBA debugger combinedwith Visual Scriptdebugger.

eTest Suite—YesObjects are recognized atthe DOM level.

eTest Suite—YesInterface to call anyexternal program or .dll

eTest Suite—YesCapability is called e-PI

eTest Suite—Yes

eTest Suite—YesWizard driven

eTest Suite—No

eTest Suite—Yes

Table B.1 Test Tool Characteristics: Capture/Playback and Script (continued)



28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 250

s

R

L


Mercury Interactive toolsoffer full debuggingcapabilities.


Mercury Interactive tools—Yes

Mercury Interactive tools—YesPublished APIs.

Mercury Interactive tools donot require any modificationof the application under test.


Mercury Interactive tools—Do not support datageneration but allow forautomatic retrieval andimport of real test data.


Rational Suite TestStudio, RationalRobot—YesDebugging to include a compile option,debug run mode, go until cursor mode,step mode, breakpoints and a watchvariables window.

Rational Suite TestStudio, RationalRobot—YesObject recognition is available andcustomizable.

Rational Suite TestStudio, RationalRobot—YesExternal dll and exe files can be calledfrom within test scripts. Also a GUIscript can detect if a dll or exe isloaded. Also has file existence and filecompare verification.

Rational Suite TestStudio, RationalRobot, Rational TestManager—Yes

Rational Suite TestStudio, RationalRobot—YesSpecific environments like Delphirequire code insertion (running theEnabler utility) to achieve objectrecognition.

Rational Suite TestStudio, RationalRobot—YesBoth GUI and VU scripts can beaugmented to handle data variation.

Rational Suite TestStudio, RationalRobot—YesDatapools can be created for use withGUI or VU scripts.


SilkTest—Yes

SilkTest—YesStrong feature of SilkTest.

SilkTest—Yes

SilkTest—YesExtension Kit allows directcommunication with app.

SilkTest—YesTool is not invasive.

SilkTest—YesObject oriented featureimplemented to supportdata driven testing.

SilkTest—Yes

SilkTest—Yes

Table B.1 Test Tool Vendors: Capture/Playbackand Script Creation (continued)



(continued )

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 251

S

R

L


Allows for adding comments duringrecording

Allows for automatic or specifiedsynchronization between clientand server

Allows for verification of objectpropertiesAllows for object data extraction andverification

Allows for database verification

Allows for text (alphanumeric)verification

Allows for wrappers (shells) wherebymultiple procedures can be linked andcalled from one procedure

Allows for automatic data retrieval fromany data source—RDBMS, legacysystem, spreadsheet—for data-driventesting

Allows for use of common spreadsheetfor data-driven testing



TestPartner—Yes


QACenter—Yes

QACenter—Yes

TestPartner—Yes


eTest Suite—Yes

eTest Suite—Yes

eTest Suite—YesAll properties of DOMobjects

eTest Suite—YesVia ODBC functions in VB.

eTest Suite—Yes

eTest Suite—Yes

eTest Suite—YesVia ODBC in scripts ordump to .csv file formatto populate a databank

eTest Suite—YesSupports .csv format




28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 252

s

R

L





WinRunner—YesVia ODBC and nativedatabase connectivity.Astra QuickTest—Yes viaVBScript functions.



WinRunner—Yes

WinRunner—YesAstra QuickTest—YesBoth support Excel format.

Rational Suite TestStudio, RationalRobot—YesComments may be added duringrecording.

Rational Suite TestStudio, RationalRobot—YesWait States for GUI playback For VU scripts, synchronization isautomatic.


Rational Suite TestStudio, RationalRobot—YesVia ODBC functions in SQABasic


Rational Suite TestStudio, RationalRobot, Rational TestManager—YesUses either shell scripts (Robot),LoadTest v2000 schedules,TestManager v2001 Suites.

Rational Suite TestStudio, RationalRobot—YesVia ODBC in scripts or dump to .csvfile format. SQABasic can also read.csv files directly. Also can use .csvfiles to create datapool source data.

Rational Suite TestStudio, RationalRobot—YesUses .csv format

SilkTest—Yes

SilkTest—YesCan automatically connectto a particular agent, oruse a script to connect.Can change from oneparticular agent to another.

SilkTest—Yes

SilkTest—YesVia database Tester 4Test functions.

SilkTest—Yes

SilkTest—Yes

SilkTest—YesDatabase Tester functionsused to supply an ODBChookup, or can scriptparsing for any files.

SilkTest—Yes.Scripts can access andmanipulate content withinall common spreadsheets.




(continued )

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 253

S

R

L


Ease of maintaining script whenapplication changes


eTest Suite—YesAutomatic update ofscripts, no recodingrequired.




Test Tool Database

Tool database scalability when testrepository grows

Network-based test repository,necessary when multiple access torepository required

Supports ANSI SQL execution

NA

QACenter—Yes

TestPartner—YesPerformance Edition—Yes

eTest Suite—YesSupports MS Access,MS SQL Server, andOracle

eTest Suite—YesShared workspaces

eTest Suite—YesSQL execution can bedone through VB.

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 254

s

R

L


WinRunner—YesAstra QuickTest—YesVia GUI map or objectrepository

Rational Suite TestStudio, RationalTeamTest, Rational Robot—YesRobot editor and debug tools enableeasy maintenance.

SilkTest—Yes.Include file can be updatedwhenever objects change in nameor nature.




Test Tool Database

WinRunner & Astra QuickTestdo not use a database.TestDirector support for allmajor database vendors.

TestDirector—Yes

WinRunner—Yes

Rational Suite TestStudio—YesFor v2001—database repositoryreplaced with an XML—baseddatastore. Within the datastore, Accessdatabase used for queries and reports.

Rational Suite TestStudio—Yes

Rational Suite TestStudio, RationalRobot—YesVia SQL commands in SQABasic.

SilkTest—Yes.Doesn't have limitations orslowdowns based on the sheervolume of test code.

SilkTest—Yes.Files can be placed anywhere,provided there is networkdirectory access to test code.

SilkTest—Yes

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 255

S

R

L


Ability to kick off scripts at a specifiedtime; scripts can be scheduled and rununattended

Has a complete graphical palette todesign and to schedule test runs basedon dates and time, or logic betweentests

Stateless playback: Application doesnot have to be in a preset state beforescript playback

Centralized execution and control

Standalone test execution automation

QACenter—YesPerformance Edition—YesQADirector allows scheduling forspecific dates/times and /or regulartime periods.

PE/ QADirector do not havegraphical palette.

Inclusion of File—AID/CS allows PEto test application DBMS states.QADirector allows for setup,execution and restore

QADirector supports access,execution, review of results from a central location.


eTest Suite—YesSchedules Visual Scripts orsets of scripts to rununattended.

eTest Suite—Yes

eTest Suite—Yes

eTest Suite—Yese-Test console providescentralized monitoring ofall application activity.

eTest Suite—Yes

Table B.2 Test Execution Capability


Test Control Features

Distributed Test Execution

Distributed test control,synchronization, execution

Execution of manual orautomated tests via a browserfor anywhere/anytime test execution,with Web-based test management

Supports synchronization of multitestthreads

QACenter—YesQADirector supports distributedexecution where app is tested as itruns in productionPerformance Edition—Yes

Manual test execution is web-based. Web interface is underwayfor other features.

For QADirector tests can be parallelor singularPerformance Edition—Yes

eTest Suite—Yes

eTest Suite—No

eTest Suite—Yes

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 256

s

R

L

257B.2 Test Execution Capability

TestDirector—allowsscheduling.LoadRunner has built-inschedulers for loadscenarios.

TestDirector—Yes

WinRunner—Yes

TestDirector—Yes


SilkTest—Yes No separate tool required

SilkTest—No

SilkTest—YesAny script can be run at any time

SilkTest—Yes

SilkTest—Yes



Test Control Features


TestDirector—allows forlocal or remote test executioncontrol via networks.

TestDirector—Yes. All partsof TestDirector can beaccessed via a browser.

WinRunner—Yes

Rational Suite TestStudio, RationalTestManager—Yes

Rational Suite TestStudio—Notcurrently

Rational Suite TestStudio—Yes

SilkTest—Yes

SilkTest—No No web front end at present. Onecould be created, since tool canrun from the command line.

SilkTest—Yes

(continued )

Rational Suite TestStudio, RationalRobot—YesRobot script checks the time, LoadTestv2000 & TestManager v2001 scheduledto run at a specific time.

Rational Suite TestStudio, RationalRobot—YesSuite creation in TestManager v2001 isall graphical.

Rational Suite TestStudio, RationalRobot—YesRobot GUI scripts can locate anywindow, by using window set context tolocate a window to begin scriptplayback. VU scripts are completelystateless.



28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 257

Table B.2 Test Execution Capability (continued)



S

R

L


Headless back-end server testing

Allows for test execution across LANsand WANs

Test results analysis: automaticcreation of test results log

Multiplatform testing support

Performance Edition—YesQADirector tests can be routed toany server or workstation attachedto the network.

Performance Edition—YesQADirector supports a centralrepository that can be accessed viathe network.

QACenter—YesQADirector supports both pass/failas well as drill down.Performance Edition—Yes

Performance Edition—YesQADirector supports Unix, Windowsand OS/390

eTest Suite—Yes

eTest Suite—Yes

eTest Suite—Yes

eTest Suite—YesSupports Unix and variousWindows.

Test Suite Recovery Logic

Unexpected error recovery: Can handleerror recovery and unexpected activewindows, logs the discrepancy, andcontinues playback (automaticrecovery from errors)

Can restore the application to its initialstate or provides ways for theapplication to be restored


QACenter—YesPerformance Edition—YesQADirector—Yes

eTest Suite—Yes

eTest Suite—Yes

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 258

s

R

L


WinRunner—Yes

TestDirector—Yes

WinRunner—YesAstra QuickTest—YesVisually displayed for ease ofscanning.

Mercury Interactive toolsprovide support for over 28different platforms.



Rational Suite TestStudio, RationalTestManager—YesTest results are viewed fromTestManager. Defects andenhancements may be reported directlyfrom the Test Log via ClearQuestintegration.

Rational Suite TestStudio, RationalTestManager—YesSupports Unix and various Windows

SilkTest—Yes

SilkTest—YesNetwork connectivity needed tothe particular machine with theSilkTest agent.

SilkTest—Yes

SilkTest—Yes




Test Suite Recovery Logic

WinRunner—YesAstra QuickTest—YesVia exception handler.


Rational Suite TestStudio, RationalRobot—YesError recovery for script failures andverification point failure. Alsounexpected active window handler canbe set. When fatal error is encountered,trap feature can be used to obtain astack trace, module lists, restartWindows or run a procedure.


SilkTest—Yes

SilkTest—Yes

(continued )

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 259



Test Management

S

R

L


Supports test execution management

Supports industry standards in testingprocess (SEI/CMM, ATLM, ISO)

Application requirements managementsupport integrated with the testmanagement tool

Requirements management capabilitysupports the trace of requirements totest plans to provide requirementcoverage metrics

Test plans can be importedautomatically into test managementrepository from standard text files

Tool can be customized toorganization’s test process

Supports planning, managing,and analyzing testing efforts;can reference test plans,matrices, product specifications,in order to create traceability

Supports manual testing

Supports the migration from manual toautomated scripts



QADirector—Integrates withReconcile/DOORS for RMPerformance Edition—NA

QADirector Integrated withReconcile—Yes

QADirector—Yes Performance Edition—NA

QADirector supports various testingmethodologies.Performance Edition—NA

QADirector—plan and report testactivity.Performance Edition—NA

QADirector—provides manual testcreation.Performance Edition—NA

QADirector—can replace manualtests with automated testing scriptsPerformance Edition—NA

eTest Suite—YesThrough e-Manager

eTest Suite—Yes

eTest Suite—No

eTest Suite—No

eTest Suite—No

eTest Suite—Yes

eTest Suite—Yese-Manager and d-Trackerprovide the ability to trackand manage the testingprocess

eTest Suite—Yes

eTest Suite—No

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 260

s

R

L


TestDirector—Yes

TestDirector—Yes

TestDirector—has built-inrequirements management.Also integrates with 3rd partyRM tools like TBI Caliber,DOORS and RequisitePro

TestDirector—Yes

TestDirector—Yes

TestDirector—Yes

TestDirector—Yes

TestDirector—Yes

TestDirector—YesGenerates commented scriptfor ease of recordingbusiness processes.


Rational Suite TestStudio, RationalTeamTest, Rational TestManager,Rational Unified Process—Yes

Rational Suite TestStudio, RationalRobot—YesTestManager can utilize RequisitePro orRational Rose. Other RM tools used viathe creation of Test Input Adaptersusing TestManager’s open API.

Rational Suite TestStudio, RationalRobot—YesRequirements can be traced to TestCases within a Test Plan inTestManager.

Rational Suite TestStudio—YesVia RequisitePro integration.

Rational Suite TestStudio, RationalTeamTest, Rational TestManager,Rational Unified Process—Yes



Rational Suite TestStudio—No

SilkTest—Yes

SilkTest—Yes

SilkTest—No

SilkTest—No

SilkTest—Yes

SilkTest—YesBuilt-in test plan functionality canbe customized to process.

SilkTest—YesVia use of extensive results fileswith collapsability features andthrough using test plans.

SilkTest—YesSpecify manual tests via a testplan or usingAgent.Display() methodwithin a script.

SilkTest—YesCan replace manual tests withscripts to perform steps for you.



Test Management

(continued )

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 261



Test Management

S

R

L


Can track the traceability of tests totest requirements

Has built-in test requirements modules

Tool can check for duplicate defectsbefore logging newly found defects

Allows for measuring test progress

Allows for various reporting activities

Allows for tracking of manual andautomated test cases

QADirector—Integrates withReconcile/DOORS for RMPerformance Edition—NA

Reconcile provides full RMPerformance Edition—NA

QADirector—YesPerformance Edition—NA

QADirector—YesPerformance Edition—NA

QADirector—YesAlso supports 3rd party reportgenerators & reports to the webPerformance Edition—NA

QADirector—YesAlso provides file/data mgmt Performance Edition—NA

eTest Suite—No

eTest Suite—No

eTest Suite—Yes

eTest Suite—Yes

eTest Suite—Yes

eTest Suite—No

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 262

s

R

L


TestDirector—Yes

TestDirector—YesAlso integrates with 3rd partyRM tools

TestDirector—Yes

TestDirector—Yes

TestDirector—Yes

TestDirector—Yes

Rational Suite TestStudio, RationalTestManager—Yes viaTestManager coverage reports.

Rational Suite TestStudio, RationalTestManager—YesVia RequisitePro

Rational Suite TestStudio, RationalClearQuest—Yes


Rational Suite TestStudio, RationalTeamTest, Rational TestManager—Yes

Rational Suite TestStudio, RationalTestManager—YesTestManager is 100% test case centricto allow for manual & automatedimplementation.

SilkTest—Yes

SilkTest—No

SilkTest —NA Does not support defect tracking.

SilkTest—Yes

SilkTest—Yes

SilkTest—Yes



Test Management

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 263

S

R

L


Interface to softwarearchitecture/modeling tool

Integration to unit testing tools

Interface to test management tool

Interface to requirements managementtool

Interface to defect tracking tool

Interface to configuration managementtool

Does not have direct integration

QADirector—YesWith TestPartner, TrueTime,ActiveAnalysis, TrueCoverage,BoundsChecker.

QADirector—YesWith Reconcile & TrackRecord

QADirector—YesWith Reconcile & DOORS

QADirector—YesWith TrackRecord

QADirector—YesVersion control tests within CM toolof choice

Does not have directintegration

eTest Suite—Supports unittesting of MiddlewareObjects.

eTest Suite—Yes

eTest Suite—No

eTest Suite—Yes

eTest Suite—No

Table B.3 Tool Integration Capability


28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 264

s

R

L

265B.3 Tool Integration Capability


Mercury Interactive tools—YesSuch as McCabe tools.

Mercury Interactive tools—YesWinRunner, LoadRunner,Astra QuickTest and AstraLoadTest integrate withTestDirector.

TestDirector—YesIntegrates with 3rd party RMtools

Mercury Interactive tools—YesBuilt-in defect managementtool. Also integrate withClearQuest and PVCS Tracker

TestDirector—YesBuilt-in interface to MSSourcesafe, PVCS VersionManager and ClearCase

Rational Suite Enterprise, Rational SuiteDevelopmentStudio, Rational SuiteTestStudio, Rational TestManager—YesRational Rose model elements can beused as test inputs using TestManagerv2001. Can generate scripts from UMLmodels in Rose.

Rational Suite Enterprise, Rational SuiteDevelopmentStudio—YesRational Quality Architect v2001 (EJB &COM)


Rational Suite TestStudio, RationalTestManager—YesTestManager fully integrates withRequisitePro and Rational Rose. OtherRM tools integrated via the Test InputAdapter API.

Rational Suite TestStudio, RationalTestManager—YesTest Log is integrated with ClearQuest

Rational Suite TestStudio—YesClearCase LT shipping with testingproducts v2001 and beyond

SilkTest—No

SilkTest—Unit testing supportedby SilkTest.

SilkTest—Yes

SilkTest—No

SilkTest—Yes

SilkTest—Yes



28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 265

S

R

L


Error filtering and review features

Metric collection and metric analysisvisualization


Performance Edition—YesMetric collection & reporting

eTest Suite—Yes

eTest Suite—Yes

Table B.4 Test Reporting Capability


Summary-Level Reporting

Test Report Presentation

Predefined reports can be modified andnew reports created

Most widely used views can be createdand made public for others to view attheir leisure with automatically updatedinformation

Generate graphs and reports from testresults

Graphs and reports fully customizable,supporting more than just limitedmodification, such as allowing choiceof axis labels

Reports are exportable to HTML,.csv, or .txt files

Performance Edition—YesReports modified & created

Performance Edition—YesFields & views customizable

Performance Edition—YesLibrary of graphs/reports


Performance Edition Outputs to HTML format

eTest Suite—Yes

eTest Suite—YesHTML output to web.Complete flexibility withWeb Reporter.

eTest Suite—Yes

eTest Suite—Yes

eTest Suite—Yes

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 266

s

R

L

267B.4 Tool Reporting Capability

TestDirector—Yes

TestDirector—Yes

Rational TestManager—YesUsers create Test Log filters and sortthe log items


SilkTest—Yes

SilkTest—Yes



Summary-Level Reporting

Test Report Presentation

TestDirector—YesDocument Generatorprovides full customization ofreports.

TestDirector—Yes

TestDirector—Yes

TestDirector—Yes

TestDirector—YesLoadRunner—YesAutomatic HTML reportgeneration setting

Rational TestManager—Yes





SilkTest—Yes

SilkTest—YesAbility to view work of others andcan control write-abilty throughuse of permissions.

SilkTest—Yes

SilkTest—No

SilkTest—Yes

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 267

S

R

L


All users can be queued to execute aspecified action at the same time

Automatic generation of summary loadtesting analysis reports

Ability to change recording of differentprotocols in the middle of load-recording session

Actions in a script can be iterated anyspecified number of times withoutprogramming or rerecording of thescript

Different modem connection speedsand browser types can be applied to ascript without any rerecording

Load runs and groups of users withinload runs can be scheduled to executeat different times

Automatic load scenario generationbased on load testing goals:hits/second, number of concurrentusers before specified performancedegradation, and so on

Cookies and sessions IDs automaticallycorrelated during recording andplayback for dynamically changing Webenvironments



Performance Edition—YesCan record multiple middlewareand protocols during samerecording session.






eTest Suite—Yes

eTest Suite—Yes

eTest Suite—YesCan change protocols aslong as within the webtransaction and can berecorded.

eTest Suite—YesWith or without variabledata.

eTest Suite—Yes

eTest Suite—Yes

eTest Suite—No

eTest Suite—YesNever requiresprogramming, entirelyautomatic

Table B.5 Performance Testing and Analysis Capability


Load and Stress Test Features

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM 8

s

R

L

269B.5 Performance Testing and Analysis Capability

LoadRunner—YesAstra LoadTest—Yes


LoadRunner—Yes for someprotocols.

LoadRunner—YesAstra LoadTest—YesA simple runtime setting.

LoadRunner—YesAstra LoadTest—YesA simple runtime setting.


LoadRunner—Yes

LoadRunner—YesAstra LoadTest—YesDoes not requireprogramming, entirelyautomatic.

Rational Suite TestStudio, RationalTestManager—YesVia sync points


Rational Suite TestStudio, RationalRobot—YesAll protocols can be captured during asingle recording session, so “changing”protocols is not necessary. Afterrecording, the multiple protocols can beput into a single script, or, filtering couldbe used to put a single, or acombination of multiple protocols into asingle script.

Rational Suite TestStudio, RationalRobot, Rational TestManager—YesMost other tools require that all useractions be put in a single script———Rational Robot allows a user session tobe split into multiple scripts, each ofwhich can be iterated any number oftimes via the graphical schedulingmechanism in TestManager.



SiteLoad, Rational Suite TestStudio—Yes (in TestManager via Transactors)

Rational Suite TestStudio, RationalTestManager, Rational Robot—Yes

SilkPerformer—Yes

SilkPerformer—Yes

SilkPerformer—Yes

SilkPerformer—Yes

SilkPerformer—Yes

SilkPerformer—Yes

SilkPerformer—Yes

SilkPerformer—Yes




(continued )

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 269

S

R

L


Allows for variable access methods andability to mix access methods in asingle scenario: modem simulation orvarious line speed simulation

Ability to have data-driven scripts thatcan use a stored pool of data

Allows for throttle control for dynamicload generation

Allows for automatic service-levelviolation (boundary value) checks

Allows for variable recording levels(network, Web, API, and so on)

Allows for transaction breakdown/drill-down capabilities for integrityverification at the per client, persession, and per instance level forvirtual users

Allows for Web applicationserver integration: ColdFusion,NetDynamics, Dynamo, and so on

Supports workload, resource, and/orperformance modeling

Can run tests on various hardware andsoftware configurations






Performance Edition—YesOffers integrated load testing andserver monitoring.




eTest Suite—Yes

eTest Suite—Yeswith DataBank Wizard

eTest Suite—Yes

eTest Suite—Yes

eTest Suite—No

eTest Suite—Yes

eTest Suite—Yes

eTest Suite—Yes

eTest Suite—Yes

Table B.5 Performance Testing and Analysis Capability (continued)



28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 270

s

R

L




LoadRunner—Yes


LoadRunner—Yes


LoadRunner—YesAstra LoadTest—YesIncluding ability to record andreplay against theapplications, and monitor theapp server performanceduring load test.



Rational TestManager Suite— YesAny combination of tests is possible.

Rational Suite TestStudio, RationalTestManager, Rational Robot—YesVia Datapools.

Rational Suite TestStudio, RationalTestManager—YesWith Transactors (graphical point andclick creation) or Shared Variables(hand coded)







SilkPerformer—Yes

SilkPerformer—Yes

SilkPerformer—Yes

SilkPerformer—Scriptscan be deployed to supportthese checks.

SilkPerformer—YesAllows variable levels andmixing of protocols for ascript, user or transaction.

SilkPerformer—Yes

SilkPerformer—Yes

SilkPerformer—Yes

SilkPerformer—Yes




(continued )

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 271

S

R

L


Support headless virtual user testingfeature

Requires low overhead for virtual userfeature (Web, database, other?)

Scales to 500–1,000 virtual users

Simulated IP addresses for virtual users

Thread-based virtual user simulation

Process-based virtual user simulation

Centralized load test controller

Allows for reusing scripts fromfunctional test suite

Support for WAP protocol testingagainst WAP Gateway or Web server

Compatible with SSL recording


Performance Edition—YesExceptionally low overhead






Performance Edition—YesFunctional/load test scripts can becreated at same time. Two usedifferent languages.



eTest Suite—Yes

eTest Suite—YesSmall footprint per Virtual User

eTest Suite—Yes

eTest Suite—Yes

eTest Suite—YesDefault and most efficientmode of testing

eTest Suite—YesOptional

eTest Suite—Yes

eTest Suite—YesVisual Scripts createdduring functional testingreused for load testing andmonitoring

eTest SuiteYes for the web server

eTest Suite—Yes




28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 272

s

R

L



LoadRunner—YesVery high scalability ratingsusing TurboLoad technology.

LoadRunner—YesLargest load recorded to dategenerated more than 1.35Mconcurrent connections.Astra LoadTest—Yes

LoadRunner—YesAstra LoadTest—YesBoth use IPSpoofer feature.


LoadRunner—YesIncluded since some appsrequire it for accuracy.


Astra QuickTest—YesWinRunner scripts used asGUI Vusers in LoadRunner.

LoadRunner—Yes

LoadRunner—YesAstra QuickTest—Yes






Rational Suite TestStudio, RationalTestManager—YesEach virtual user has it own processspace and unimpeded access to allresources within a process. Multi-threaded model supported whenneeded.





SilkPerformer—Yes

SilkPerformer—Yes

SilkPerformer—Yes

SilkPerformer—Yes

SilkPerformer—Yes

SilkPerformer—YesCan set any number of virtualusers as a universal defaultparameter or set number withina script.

SilkPerformer—Yes

SilkPerformer—Yes

SilkPerformer—Yes

SilkPerformer—Yes




(continued )

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 273




S

R

L


Compatible with one or more of therelevant technologies: streamingmedia, COM, EJB, RMI, CORBA, Siebel,Oracle, SAP

Compatible with one or more of therelevant technologies: Linux, UNIX, NT,XWindows, Windows CE, Win3.1,Win95, Win98, Win2000, WinME



eTest Suite—Yes

eTest Suite—Yes

Performance Monitor Test Features

Monitors various tiers: Web server,database server, and app serverseparately

Supports monitoring for one or more ofColdFusion, Broadvision, BEAWebLogic, Silverstream, ATG Dynamo,Apache, IBM Websphere, OracleRDBMS, MS SQL Server, Real MediaServer, IIS, Netscape Web Server

Supports monitoring for one or more ofthe relevant technologies: Linux, NT,UNIX, XWindows, Windows CE, Win3.1,Win95/98, Win2000

Monitors network segments

Supports resource monitoring

Synchronization ability in orderto determine locking, deadlockconditions, and concurrencycontrol problems

Ability to correlate any metrics from allmonitors to identify performancebottlenecks




Performance Edition—YesWith EcoSystems and ApplicationExpert




eTest Suite—Yes

eTest Suite—Yes

eTest Suite—Yes

eTest Suite—Yes

eTest Suite—Yes

eTest Suite—Yes

eTest Suite—Yes

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 274




s

R

L


LoadRunner—Yes




SilkPerformer—Yes

SilkPerformer—Yes

(continued )
















SilkPerformer—Yes

SilkVision—YesSilkVision component providesmonitoring capability.



SilkPerformer—Yes

SilkPerformer—Yes

SilkPerformer—Yes

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 275

S

R

L


Ability to detect when events havecompleted in a reliable fashion

Ability to provide client-to-serverresponse times

Ability to provide graphical results andexport them to common formats

Ability to provide performancemeasurements of data loading





eTest Suite—Yes

eTest Suite—Yes

eTest Suite—Yes

eTest Suite—Yes




28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 276

s

R

L





LoadRunner—Yes



Rational Suite TestStudio, RationalTestManager—YesResults can be exported to .csv.


SilkPerformer—Yes

SilkPerformer—Yes

SilkPerformer—Yes

SilkPerformer—Yes




28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 277

S

R

L


Maturity of vendor

Market share of vendor

On the market since 1996

Growing market share

On the market since 1996

Growing market share

Table B.6 Vendor Qualifications


Consulting Requirements

Vendor Support

Software patches provided, if sodeemed necessary

Upgrades provided on a regular basis

Upgrades backward compatible: scriptsfrom previous version can be reusedwith later version

Training available

Help feature available; tool welldocumented

Tech support reputation throughoutindustry

Yes

YesMajor releases once per year, pointreleases once per quarter, bug fixesas necessary.

Yes

YesOnsite and public training.

Yes

Highly rated technical support.

Yes

YesMajor releases once peryear, point releases onceper quarter, bug fixes asnecessary.

Yes

YesOnsite and publictraining.

Yes

Highly rated technicalsupport from customerservice surveys.

Vendor Qualifications

Financial stability of vendor

Time existed

Profitable company

Since 1973

Former subsidiary ofTeradyne. Formed by thecombination of HammerTechnologies and RSWSoftware and funded byMatrix Partners.

Since 1996

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 278

Continued/consistent growthpattern over last severalyears.

Company incorporated in1989

s

R

L

279B.6 Vendor Qualifications

Tools have been supportingindustry since early 1990s.

Considerable market share

Various testing products have been onthe market since early 1980s.

Sustaining market share

SilkTest/QA Partner products onthe market since early 1990s.

Sustaining market share



Consulting Requirements

(continued )

Vendor Support

Yes

YesBuilt-in updater in theproduct to check for newpatches and features via theInternet.

Yes

Onsite and public training.Public training facilitiesaround the world.

Yes

Given prestigious SSPA STARaward last two years forexcellence in support.

Yes

YesEvery six months.

YesUpgrade and conversion utilities arealways provided.

YesRational UniversityOnsite and public training.

Yes

YesService has received many awards

Yes

Yes

YesAlways within major versionreleases.

Yes

Yes

Good Tech support.

Vendor Qualifications

Stable/profitable company

Since early 1980s

Stable/profitable company

Since early 1990s

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 279

S

R

L


No consulting needed

Availability of and access to tool usergroups

Consulting not needed

Several discussion forums

Consulting not needed


Table B.6 Vendor Qualifications


Vendor Support

Pricing

Price consistent within estimated pricerange

Price consistent with comparablevendor products

NA

Yes

NA

Yes

Licensing

Allows for floating license

Allows for node-locked license

Licensing used

Licensing rigid

Floating and node-locked licenses.

Yes

Yes

Flexibility is available.


Yes

Yes

Flexibility is availableand is a commercialdecision

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 280

s

R

L

281B.6 Vendor Qualifications

Consulting not needed forAstra QuickTest or AstraLoadTest. A QuickStart isrecommended for others toobtain maximum return ontool investment.

Complete network of localusers groups throughoutworld.

Consulting generally not needed.Depends on type of testing to beperformed.


None needed. Jump startprograms available andrecommended for teams withtight time constraint.

User Groups exist.



Vendor Support

Pricing

NA

Yes

NA

Yes

NA

Yes

Licensing

Completely flexible licensingfrom pay-per-use toengagement based toperpetual pricing

Yes

Yes

Completely flexible licensing


Yes

Yes

Yes

Floating and node-lockedlicenses.

Yes

Yes

No. Licensing can be based onmost need-based models forcustomers.

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 281

28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 282

Test Tool Evaluations - UML Testing Tool Evaluation Matrix.pdf · 248 Appendix B Test Tool...

Documents

Transcript of Test Tool Evaluations - UML Testing Tool Evaluation Matrix.pdf · 248 Appendix B Test Tool...