Fast matrix computations for pair-wise and column-wise Katz scores and commute times
Test Smarter (with Pair-wise Testing)
-
Upload
softwarecentral -
Category
Documents
-
view
7.130 -
download
0
Transcript of Test Smarter (with Pair-wise Testing)
Test Smarter(with Pair-wise Testing)
Presented by: Bill LewisCEO, Smartware Technologies, Inc.
(469) 951-7548www.smartwaretechnologies.com
Copyright 2010 Smartware Technologies, Inc. 2
Agenda
• Challenges of application development• Automated Testing• What is orthogonal arrays?• Power of pair-wise testing• Transforming theory to practice• Summary• Q&A
Copyright 2010 Smartware Technologies, Inc. 3
“V” Model
Coding
Unit TestingDesign
Integration TestingArchitecture
System TestingFunctional &NF Requirements
UA TestingUser Requirements
Project SuccessBusinessRequirements
Static TestingVerify
Dynamic TestingValidate50 % 50 %
Copyright 2010 Smartware Technologies, Inc. 4
Tractability Matrix• A Trace Matrix can be used to track test cases
during development and execution
– Identification of uniquely “testable” requirements• Eliminate wishes, hopes, dreams, and other un-testable
statements
• Eliminate duplicates
• Assign a unique ID for each requirement and design objective
• Determine its priority if possible
– Verification of implementation of all requirements
Copyright 2010 Smartware Technologies, Inc. 5
Traceability Matrix Example
Copyright 2010 Smartware Technologies, Inc. 6
Issues Facing Software Testing Organizations
• Today’s software is getting more and more complex with thousands if not millions of different combinations that need to be tested.
• Humans can’t think of all the test scenarios• Takes too much time• Too error prone• Which scenarios do you test?
• The length of time to test modern software is increasing.
• Requirements documentation is often ambiguous or inadequate
• Testing process is poorly defined or non-existent
Copyright 2010 Smartware Technologies, Inc. 7
Simple Example: Configuration Testing
Copyright 2010 Smartware Technologies, Inc. 8
Simple Example (Con’t)• Suppose you have an application that needs to be tested
on several hardware and software configurations.
• You are told that the application must be able to run on a Windows NT, Windows 2000 and Windows XP machine.
• It also has to work on a system with a minimum of 128Meg and a max of 512Meg of RAM.
• Finally, the application must also run on a Pentium II, III, and IV processor and work with Oracle, SQL and Access Databases.
Copyright 2010 Smartware Technologies, Inc. 9
Simple Example (Con’t)
Operating System
RAM PC Processor
Database
Windows NT 128 Meg Pentium II Oracle
Windows 2000
256 Meg Pentium III SQL
Windows XP 512 Meg Pentium IV Access
Determine test cases for testing the an application on the following:
Copyright 2010 Smartware Technologies, Inc. 10
Combinations of Test Cases3 x 3 x 3 x 3 = 81 Test Cases
Copyright 2010 Smartware Technologies, Inc. 11
Why Automate?
• Over time, number of functional features changes/ increase
and• Number of people and
amount of testing time for each new release
– Remains flat– May even decline
• Test coverage steadily decreases
– Increases the risk of failure– Potential business losses
Source: Linda Hayes, “Why Certify”
Copyright 2010 Smartware Technologies, Inc. 12
• Latest generation of automated testing tools support variable test data
• Difference between static and variable capture/ replay – With Static the inputs and outputs are fixed– With variable inputs and outputs are variable– Replacing the captured inputs and expected outputs
with variables – Variable capture/ replay reduces the risk of not performing
regression testing on existing/ new features
Today’s Test Automation Variable Capture/ Replay Tools
Copyright 2010 Smartware Technologies, Inc. 13
Test Automation Framework
Source: Ross CollardSource: Ross Collard
What’s the Missing Link?
Answer: scientifically generated test data
Copyright 2010 Smartware Technologies, Inc. 15
Simple Example Revisited
Operating System
RAM PC Processor
Database
Windows NT 128 Meg Pentium II Oracle
Windows 2000
256 Meg Pentium III SQL
Windows XP 512 Meg Pentium IV Access
Determine test cases for testing the an application on the following:
* Constraint: Windows XP cannot run on a Pentium II PC Processor.
Copyright 2010 Smartware Technologies, Inc. 16
Optimal Test Cases(via Pair-Wise Analysis)
Test Case #
Operating System
RAM PC Processor Database
1 Windows NT 128 Meg Pentium II Oracle
2 Windows NT 256 Meg Pentium III SQL
3 Windows NT 512 Meg Pentium IV Access
4 Windows 2000 128 Meg Pentium III Access
5 Windows 2000 256 Meg Pentium IV Oracle
6 Windows 2000 512 Meg Pentium II SQL
7 Windows XP 128 Meg Pentium IV SQL
8 Windows XP 512 Meg Pentium III Oracle
Copyright 2010 Smartware Technologies, Inc. 17
Pairwise Definition
Pairwise testing – an approach to combinatorial testing that executes a pairwise test data set.
Pairwise test data set - A set of test cases that covers all combinations of the selected test data values for every pair of a system’s input variables.
Pairwise testing is also known as 2-way testing; other approaches include 3-way, 4-way, … , n-way.
Copyright 2010 Smartware Technologies, Inc. 18
Secret Sauce(Orthogonal Arrays)
Mathematical curiosity discovered by monks in the 1400s
Promoted by Dr. Genichi Taguchi in the 1950s for use in statistical test design in manufacturing, agriculture, medicine, etc.
Orthogonal Array: Two dimensional array of numbers in which any two columns in the array receive an even distribution of all pairwise combinations of values in the array.
• Orthogonal arrays are a standard construction used for statistical experiments.
Copyright 2010 Smartware Technologies, Inc. 19
Example Orthogonal Arrays and Notation1 2 3 4 5 6 7
1 0 0 0 0 0 0 0
2 0 1 2 2 0 1 1
3 0 2 1 2 1 0 2
4 0 1 1 0 2 2 3
5 0 2 0 1 2 1 4
6 0 0 2 1 1 2 5
7 1 1 1 1 1 1 0
8 1 2 0 0 1 2 1
9 1 0 2 0 2 1 2
10 1 2 2 1 0 0 3
11 1 0 1 2 0 2 4
12 1 1 0 2 2 0 5
13 2 2 2 2 2 2 0
14 2 0 1 1 2 0 1
15 2 1 0 1 0 2 2
16 2 0 0 2 1 1 3
17 2 1 2 0 1 0 4
18 2 2 1 0 0 1 5
1 2 3
1 0 0 0
2 0 1 1
3 1 0 1
4 1 1 0
L4 (23)
1 2 3 4
1 0 0 0 0
2 0 1 1 1
3 0 2 2 2
4 1 0 1 2
5 1 1 2 0
6 1 2 0 1
7 2 0 2 1
8 2 1 0 2
9 2 2 1 0
L18(3661)
L9(34)
Copyright 2010 Smartware Technologies, Inc. 20
Why is Pair-wise Testing So Efficient?
• Example: Insurance Application (GUI)
• Suppose a bug exists wherein the application produces a garbled record when the spouse's name is empty, but only when the "married" radio button is selected
• With pair-wise testing we’ll find the bug!
Dependents (checkbox)
Marital Status
(radio buttons)
Spouse First Name (text
field less or equal 20
characters)
No Children Single Mary
Dependent Children
Married Greater than 20 characters
Adult Children
Divorced Empty
Copyright 2010 Smartware Technologies, Inc. 21
Exhaustive TestingNo Children Single Mary
No Children Single Greater than 20 characters
No Children Single Empty
No Children Married Mary
No Children Married Greater than 20 characters
No Children Married Empty
No Children Divorced Mary
No Children Divorced Greater than 20 characters
No Children Divorced Empty
Dependent Children
Single Mary
Dependent Children
Single Greater than 20 characters
Dependent Children
Single Empty
Dependent Children
Married Mary
Dependent Children
Married Greater than 20 characters
… 3 x 3 x 3 = 27 tests
Copyright 2010 Smartware Technologies, Inc. 22
Pair-wise Results
When “business rules” (functional coverage) are also applied to the results, the probability of detecting a defect is
dramatically increased.
(Only 9 test cases are required)
Copyright 2010 Smartware Technologies, Inc. 23
Real Results of Pair-wise Testing
• Identify a minimum set of tests that will find all multi-mode defects (particularly serious defects) Clarification
• All single-mode defect will be found if every option is tested at least once (unit elements)• 2-way or “Pair-wise” is combinations of 2 items (parameters) that cause a defect • 3-way or “Tri-wise” is combinations of 3 items (parameters) that cause a defect
“Pair-wise” defect detection (pair-wise testing) finds most defects, ex. testing all pairs typically finds 75% of defects Source: Kuhn
NASA Deep Space Mission - study showed that 88% of bugs discovered using “pair-wise” defect detection testing
Source: 27th NASA/ IEEE Software Engineering Workshop, NASA Goddard Space Flight Center, 4-6 Dec, 2002.
U.S. Food and Drug Administration – study showed 98% bugs discovered using “pair-wise” defect detection Source: 27th NASA/ IEEE Software Engineering Workshop, NASA Goddard Space Flight Center, 4-6 Dec, 2002.
“The Combinatorial Design Approach to Automatic Test Generation” Source: IEEE, Software 1996, Vol. 13, No. 5
• Baseline application under study resulted in a reduction in test plan time from 1 month to less than 1 week
• In several experiments, “pair-wise” demonstrated good code coverage and defect detection ability
Managing the Test Combination Explosion
Transforming Theory to Practice
Copyright 2010 Smartware Technologies, Inc. 26
How does Smart Test™ work?
Copyright 2010 Smartware Technologies, Inc. 27
Demonstration
Invoke Smart Test™ Generator
Copyright 2010 Smartware Technologies, Inc. 29
Smart Test™ Generated Test Data
QTP Script Ready to Executed
Copyright 2010 Smartware Technologies, Inc. 31
Agenda
• Challenges of application development
• Evolution of software testing• What is orthogonal arrays?• Power of pair-wise testing• Transforming theory to practice• Summary• Q&A
Copyright 2010 Smartware Technologies, Inc. 32
Questions ?
Copyright 2010 Smartware Technologies, Inc. 33
Contact Information
• If you want to contact me– Bill Lewis, CEO & President Smartware Technologies, Inc.– www.smartwaretechnologies.com– (469) 951-7548