Section 10.3: Large-Sample Hypothesis Tests for a Population Proportion.
John Kent - Europe's Largest Software Testing …...‘scripted’ tools e.g.WinRunner, QARun, SQA...
Transcript of John Kent - Europe's Largest Software Testing …...‘scripted’ tools e.g.WinRunner, QARun, SQA...
International Conference On
Software Testing, Analysis & ReviewNovember 19 - 23 Stockholm, Sweden
P r e s e n t a t i o n
W4Wednesday 21st November, 2001
State Of The ArtAutomation
John Kent
Wednesday 21 November 2001
W4
State of the Art Automation John Kent John Kent is a leading consultant specialising in test automation and test management. He writes two columns for Professional Tester Magazine: The View From Kent and Technology for Testers. He has a special interest in the Web and is co-author of the Official Netscape Guide to JavaScript1.2 (Netscape Press, 1997). John regularly presents to international audiences on a range of software testing subjects including his own training course: Using Automated Test Tools.
John Kent M.Sc. John Kent M.Sc. CISS LtdCISS Ltd
Tel: 01483 720500Tel: 01483 720500
Email: [email protected]: [email protected]
Web: http://www.cisstest.comWeb: http://www.cisstest.com
© CISS Ltd 2001
State of The Art State of The Art AutomationAutomation
nn Overview of Test AutomationOverview of Test Automation
nn Current Automated Test MethodsCurrent Automated Test Methods
nn Reasons For New ApproachesReasons For New Approaches
nn State of The Art AutomationState of The Art Automation–– Advanced ArchitecturesAdvanced Architectures
–– Test Code Creation MethodsTest Code Creation Methods
–– Reusable Test ObjectsReusable Test Objects
–– Test Process IntegrationTest Process Integration
ee
ContentsContents
© CISS Ltd 2001
nn Automated Functional Testing using Automated Functional Testing using ‘scripted’ tools e.g.WinRunner, QARun, ‘scripted’ tools e.g.WinRunner, QARun, SQA RobotSQA Robot
nn Aiming at large regression tests Aiming at large regression tests -- large large number of automated testsnumber of automated tests
nn Use automated Testing to increase Use automated Testing to increase ‘testedness’ of system‘testedness’ of system
nn Integration into test management toolsIntegration into test management tools
© CISS Ltd 2001
ee 1. Overview of Test 1. Overview of Test AutomationAutomation
ee Overview of Test Overview of Test AutomationAutomation
LogLog
Test Management Test Management ToolTool
Step01Step01 PassPass
Step02Step02 FailFail
Step03Step03 PassPass
Step04Step04 PassPass
Step05Step05 PassPass
Test ExecutionTest Execution
SUTSUT
Test01 Add CustTest01 Add Cust
Test AutomationTest Automation
nn Record/PlaybackRecord/Playback
nn Program/PlaybackProgram/Playback––Basic Data DrivenBasic Data Driven
––Advanced ArchitectureAdvanced Architecture
© CISS Ltd 2001
ee 2. Current Approaches 2. Current Approaches to Test Automationto Test Automation
1 Record/Playback1 Record/Playback
© CISS Ltd 2001
2 Data Driven2 Data Driven
ee Test Automation Test Automation Maturity ModelMaturity Model
BeginnersBeginners
Most Automation ProjectsMost Automation Projects
Program/PlaybackProgram/Playback
Manual Test
Procedures
Add customer “Bloggs”
Create an order for product X
Translated into test
tool language
Automated Test Code
Menuitem
”customer”
Type(“Bloggs”)
.....
Menuitem
”Order”
© CISS Ltd 2001
ee
Record/PlaybackRecord/Playback
TestsTests
“Bloggs”, “John”
DataDataFileFile
Test ProgramsTest Programs
Navigates to screen/window and inputs data
© CISS Ltd 1999
ee
Basic DataBasic Data--DrivenDriven
nn Automation not fulfilling its promiseAutomation not fulfilling its promise
nn Current architectures not deliveringCurrent architectures not delivering
nn Importance of MaintenanceImportance of Maintenance
© CISS Ltd 1999
ee 3. Reasons for New 3. Reasons for New ApproachesApproaches
nn Test data is hardTest data is hard--codedcoded
nn SUT Behaviour is hardSUT Behaviour is hard--codedcoded
nn Test Procedure: test program ratio =1:1Test Procedure: test program ratio =1:1
nn Many test programs to maintainMany test programs to maintain
nn Many test programs to testMany test programs to test
nn Becomes victim of own success: the Becomes victim of own success: the more tests, the more test programs to more tests, the more test programs to maintain and testmaintain and test
© CISS Ltd 1999
ee Disadvantages of Disadvantages of Record/PlaybackRecord/Playback
nn Recording does not workRecording does not work
nn Recording produces unmaintainable Recording produces unmaintainable codecode
nn We are interested in what We are interested in what maymayhappen, not what happen, not what hashas happenedhappened
nn An automated test should be an An automated test should be an interactioninteraction between the tests and between the tests and System Under Test(SUT)System Under Test(SUT)
© CISS Ltd 2001
ee Disadvantages of Disadvantages of Record/Playback(Cont)Record/Playback(Cont)
nn Most SUT Behaviour is still hardMost SUT Behaviour is still hard--codedcoded
nn Test Analysts are not free to test what Test Analysts are not free to test what they like they like -- totally business scenario totally business scenario drivendriven
nn Repetitive data fed into system Repetitive data fed into system -- not not like a testlike a test
© CISS Ltd 1999
ee Disadvantages of Disadvantages of Basic Data DrivenBasic Data Driven
nn DevelopedDeveloped
nn TestedTested
nn MaintainedMaintained
Test Automation Code Test Automation Code Must Be:Must Be:
© CISS Ltd 1999
ee Automated Testing is Automated Testing is Software EngineeringSoftware Engineering
nn Advanced Automation ArchitecturesAdvanced Automation Architectures
nn Test Analysts EmpoweredTest Analysts Empowered
nn New Ways to Create Test Automation New Ways to Create Test Automation CodeCode
nn Reusable Test ObjectsReusable Test Objects
nn Test Process IntegrationTest Process Integration
© CISS Ltd 1999
ee 4. State Of The Art 4. State Of The Art AutomationAutomation
““The software architecture of a program The software architecture of a program or computing system is the structure or or computing system is the structure or structures of the system, which structures of the system, which comprise software components, the comprise software components, the externally visible properties of those externally visible properties of those components, and the relationships components, and the relationships among them.”among them.”
Bass, Clements, and Kazman of Carnegie Mellon’s SEI Software Architecture in Practice see .sei.cmu.edu/ata/ata init.html
© CISS Ltd 2000
ee What is a Software What is a Software Architecture?Architecture?
“...designing and specifying the overall system structure emerges as a new kind of problem. Structural issues include gross organization and global control structure; protocols for communication, synchronization, and data access; assignment of functionality to design elements; physical distribution; composition of design elements; scaling and performance; and selection among design alternatives."
Mary Shaw and David Garlan in An Introduction To Software Architecture (1993) see .sei.cmu.edu/ata/ata init.html
© CISS Ltd 2000
ee What is a Software What is a Software Architecture?Architecture?
Interface Interface ModulesModules--WrappersWrappers
Add customer “Bloggs”
DataDataFileFile
Test ProgramsTest Programs
Interaction
InfrastructureInfrastructure
Drivers, Library functions, Etc..
© CISS Ltd 1999
ee Advanced Automation Advanced Automation ArchitecturesArchitectures
EXCEL
Text
csv
Test DataTest Data+Navigation+Navigation+Actions+Actions
QARun
WinRunner TSL
Test BASIC
Test Code Test Code (Programs)(Programs)
ActionsActions
© CISS Ltd 2000
ee Advanced Automation Advanced Automation ArchitecturesArchitectures
Advanced Architectures seek to move the navigation and test actions out of the test
tool code into the test data
© CISS Ltd 2000
ee Advanced Automation Advanced Automation ArchitecturesArchitectures
Advanced Architectures seek to move the navigation and test actions out of the test
tool code into the test data
This empowers the Test Analyst by allowing them to choose what the test actions are
Does not hide test actions by automation code making decisions on the fly
nn Business Object LevelBusiness Object Leveluu Business tasks are automatedBusiness tasks are automated
uu Tests specified at business language levelTests specified at business language level
uu Test Programs are ‘wrappers’ for business Test Programs are ‘wrappers’ for business tasks tasks -- must be codedmust be coded
uu Not all actions/navigation removed from codeNot all actions/navigation removed from code
© CISS Ltd 1999
ee Advanced Automation Advanced Automation ArchitecturesArchitectures
Two Types of Advanced Architecture:Two Types of Advanced Architecture:
nn Screen/window Level ArchitecturesScreen/window Level Architecturesuu Screen/windows are ‘wrapped’ by test Screen/windows are ‘wrapped’ by test
programs. Test program/screen ratio = 1:1programs. Test program/screen ratio = 1:1
uu UI ObjectUI Object--toto--UI Object Reference ratio of 1:1UI Object Reference ratio of 1:1
uu All actions/navigation removed from codeAll actions/navigation removed from code
uu Test Analysts given complete control to create Test Analysts given complete control to create the tests they wantthe tests they want
uu Can use new ways to create automation code Can use new ways to create automation code by analysing the SUT windowsby analysing the SUT windows
ee Advanced Automation Advanced Automation ArchitecturesArchitectures
Two Types of Advanced Architecture:Two Types of Advanced Architecture:
1 Record/Playback1 Record/Playback
© CISS Ltd 2001
2 Data Driven2 Data Driven
3 Advanced Architecture3 Advanced Architecture
ee Test Automation Test Automation Maturity ModelMaturity Model
ATAA (CISS)TestFrame(CMG+SDT)
ATAA (CISS)TestFrame(CMG+SDT)
BeginnersBeginners
Most Automation ProjectsMost Automation Projects
Program/PlaybackProgram/Playback
CUST ADD Smith John Perregren 29 Abbey Rd
ORDER ADD Widget Green 2inch
CUST UPDATE Smith Jon 30 Abbey Rd
CUST CHECK Smith Jon Perregren
CUST ADD B;loggs Jonhansen CUST-01
nn Actions in Test DataActions in Test Data
nn Navigation in Test DataNavigation in Test Data
nn WrappersWrappers
ee Advanced Automation Advanced Automation ArchitecturesArchitectures
#Screen Main Menu ===># TEST TESTSTEP AUTOSTEP RUNDATE ACTIONKEYTESTACTIONOPTION Message# B L=3 B L=2
MGO000C1 ACCT023 1 1 Return CDBMGO000C1-01ACCT023 1 2 Return CRG#Screen CBO500F1 =>Direct Select# TEST TESTSTEP AUTOSTEP RUNDATE ACTIONKEYTESTACTIONFHUSER FHHEAD FHTIME FHDATE# O L=10 O L=50 O L=6 O L=10
CBO500F1 ACCT023 1 3 F9#Screen CBO56010 =>Title :# TEST TESTSTEP AUTOSTEP RUNDATE ACTIONKEYTESTACTIONFHUSER FHHEAD FHTIME FHDATE# O L=10 O L=50 O L=6 O L=10
CBO56010 ACCT023 1 4CBO56010 ACCT023 1 5 Return Check#Screen GPO020F2# TEST TESTSTEP AUTOSTEP RUNDATE ACTIONKEYTESTACTIONPOSTCODEADDR1 ADDR2 ADDR3# O L=10 B L=39 B L=39 B L=39
GPO020F2 ACCT023 1 6 F10#Screen CBO56010 =>Title :# TEST TESTSTEP AUTOSTEP RUNDATE ACTIONKEYTESTACTIONFHUSER FHHEAD FHTIME FHDATE# O L=10 O L=50 O L=6 O L=10
CBO56010 ACCT023 1 7 CheckCBO56010 ACCT023 1 8 ReturnCBO56010 ACCT023 1 9 F10
#ONLINE: Create an Account# ACCT023 1
ee Empowering Test Empowering Test AnalystsAnalysts
nn For Screen/Window level Architectures For Screen/Window level Architectures OnlyOnly
nn Automatic Analysis of SUT to produce Automatic Analysis of SUT to produce model of User Interfacemodel of User Interface
nn Test Automation Code Generated from Test Automation Code Generated from model.model.
nn Generate/PlaybackGenerate/Playback
ee New Ways to Create New Ways to Create Automation CodeAutomation Code
Write Write Automation Automation ScriptsScripts
© CISS Ltd 2000
Create Create Test Test DataData
MaintainMaintainAutomation Automation ScriptsScripts
Test TestsTest Tests
ee New Ways to Create New Ways to Create Automation CodeAutomation Code
Automation Project Tasks: Program/PlaybackAutomation Project Tasks: Program/Playback
© CISS Ltd 2000
Create Create Test Test DataData
Test TestsTest Tests
Automation Project Tasks: Generate/PlaybackAutomation Project Tasks: Generate/Playback
GenerateGenerateAutomation Automation ScriptsScripts
Automatic Impact Automatic Impact Analysis &Analysis &ReRe--generate generate ScriptsScripts
ee New Ways to Create New Ways to Create Automation CodeAutomation Code
1 Record/Playback1 Record/Playback
© CISS Ltd 2001
2 Data Driven2 Data Driven
3 Advanced Architecture3 Advanced Architecture
ee Test Automation Test Automation Maturity ModelMaturity Model
ATAA (CISS)TestFrame(CMG+SDT)
ATAA (CISS)TestFrame(CMG+SDT)
BeginnersBeginners
Most Automation ProjectsMost Automation Projects
Program/PlaybackProgram/Playback 4 Mapping Methods4 Mapping Methods
5 Script Generators5 Script Generators
CISSTestCISSTest
SAG (CISS)RADStar (ICT)
SAG (CISS)RADStar (ICT)
Generate/PlaybackGenerate/Playback
nn Each Screen/window has test code Each Screen/window has test code ‘Wrapper’‘Wrapper’
nn Wrapper generated for each Screen/WindowWrapper generated for each Screen/Window
nn Layout of data for test analysts also Layout of data for test analysts also generatedgenerated
ee
Wrapper GenerationWrapper Generation
nn Test Object is a reusable piece of Test Object is a reusable piece of test datatest data
nn Can use it in other testsCan use it in other tests
nn Can overCan over--ride valuesride values
nn Puts inheritance into automated testsPuts inheritance into automated tests
ee
Test Object ReTest Object Re--useuse
nn Automated tests should integrate into Automated tests should integrate into the manual test processthe manual test process
nn Should be able to report on all tests, Should be able to report on all tests, manual and automatedmanual and automated
ee Test Process Test Process IntegrationIntegration
ee Test Process Test Process IntegrationIntegration
InfrastructureInfrastructure
Add customer“Bloggs”
DataDataFileFile
Test ProgramsTest Programs
Interaction
Drivers, Libraryfunctions, Etc..
© CISS Ltd 1999
ATAAATAA
Screen/Screen/windowwindowwrapperwrapper
LogLog
Test Management Test Management ToolTool
Step01Step01 PassPass
Step02Step02 FailFail
Step03Step03 PassPass
Step04Step04 PassPass
Step05Step05 PassPass
Test01 Add CustTest01 Add Cust
© CISS Ltd 1999
State of the art automation is about State of the art automation is about empowering the test analyst.empowering the test analyst.
Automation should be a conduit Automation should be a conduit between the tests and the system between the tests and the system under testunder test
ee State of the Art State of the Art AutomationAutomation