-Monica Kothari
description
Transcript of -Monica Kothari
-Monica Kothari-Monica Kothari
QE Efforts Optimization using
OOrthogonalrthogonal AArrayrray TTestingesting SStrategytrategy
-A Case Study
1
Execute Smartly?
2
OR
Execute Endlessly?
Major Challenges
3
Scope
What If...
4
60% Less Effort !
• Less no. of tests cases but still high functional and code coverage
• Systematic scoping of test cases across the domain with minimal manual effort
• Find new bugs which could not be found using conventional approach
• Find bugs early in the cycle
Agenda
5
Complexity of the testing problem Geometric View of Test Cases Taguchi’s OATS approach Example on Orthogonal Array Why use OATS Comparison of Test Planning Methods OATS implementation Applying the Technique ROI Analysis What works and What Doesn’t!
Complexity of a Testing ProblemComplexity of a Testing Problem
Apply Coupon(8) Verify
Total (26)SubTotal(13)Price (5)Qty (5)Ship Tax (9)Discount (5)
Regular Item (8)Store Tag Item (8)Cross-sell Item (8)Mono-Inscn Item (8)Volmn Item (8)Incr-Decr Price (8)Softgood (8)GC (8)
Full FC 15s (16)*
Full FC (16)
Temp FC (16)
Mini-cart (16)
Verify
Buyer
Apply Coupon(8) Verify
Total (26)SubTotal(13)Price (5)Qty (5)Ship Tax (9)Discount (5)
Regular Item (8)Store Tag Item (8)Cross-sell Item (8)Mono-Inscn Item (8)Volmn Item (8)Incr-Decr Price (8)Softgood (8)GC (8)
Full FC 15s (16)*
Full FC (16)
Temp FC (16)
Mini-cart (16)
Verify
Buyer
Figure3 Test Case Coverage Flow Diagram
6
Key LimitationsKey Limitations
• Generating all combinations manually is error prone:
Total Combinations = Template(5)*FCType(4)*Item(8)*Action(8)*6(Verify)
= 7,680 combinations
7
• Identifying optimal combinations to test without missing any complex combinations ensuring high functional and code coverage ensuring the confidence in quality of product.
• Minimum testing cycles for execution and automation development.
Geometric View of Test CasesGeometric View of Test Cases
8
Orthogonal Array Testing StrategyOrthogonal Array Testing Strategy
Orthogonal Array is an array of values in which:
Each column is an independent variable to be tested for interaction represented as – “Factor”.
Each Factor can take a certain set of values called “Levels”.
Each row represents a test case/combination.
Taguchi OATS ApproachTaguchi OATS Approach
Orthogonal Array Testing Strategy created by Dr. Genichi Taguchi is a proven, systematic, statistical way of testing pair-wise interactions.
9
OATS - ExampleOATS - Example A B C
1 1 1 3
2 1 2 2
3 1 3 1
4 2 1 2
5 2 2 1
6 2 3 3
7 3 1 1
8 3 2 3
9 3 3 2
TABLE1 Sample Array using OA
Total cases = 9 which cover all pair-wise combinations of the 3 variables.
Applying OATS3 Parameters – A,B,C
3 Values – 1,2,3
All possible cases involving 3 parameters:
3*3*3 = 27 cases
10
Identify a minimum set of tests that will find all multi-mode defects (particularly serious defects)
• All single-mode defect will be found if every option is tested at least once (unit elements)
• 2-way or “Pair-wise” is combinations of 2 items (parameters) that cause a defect
• 3-way or “Tri-wise” is combinations of 3 items (parameters) that cause a defect
“ Pair-wise” defect detection (pair-wise testing) finds most defects, ex. testing all pairs typically
finds 75% of defects Source: Kuhn
NASA Deep Space Mission - study showed that 88% of bugs discovered using “pair-wise” defect detection testing
U.S. Food and Drug Administration – study showed 98% bugs discovered using “ pair-wise ” defect detection Source: 27 th NASA/ IEEE Software Engineering Workshop, NASA Goddard Space Flight Center, 4-6 Dec, 2002.
“ The Combinatorial Design Approach to Automatic Test Generation” Source: IEEE, Software 1996, Vol. 13, No. 5
Baseline application under study resulted in a reduction in test plan time from 1 month to less than 1 week
In several experiments, “pair-wise” demonstrated good code coverage and defect detection ability
Why OOrthogonal rthogonal AArray rray TTesting esting SStrategy?trategy?
11
Comparison of Test Planning MethodsComparison of Test Planning MethodsTesting Method Testing
Effort (No. Of Tests)
Testing Efforts (TestPlanning)
Ability to Find Faults
Requirements Traceability (One Factor at a Time)
Small Small Limited (a)
Exhaustive Testing Very Large Small All Faults Found (100%)
Deductive/Analytical(White-Box Testing)
Medium Very Large Close to 100%
Random/Intuitive(Based on Past
History)
Medium Medium Unpredictable
Orthogonal Array-
Based Robust Testing
Small Small Better than (a); Close to 100% when used in conjunction with code coverage analysis tools.
12
Both the approaches were followed and analyzed:
1. Traditional approach to generate the test spec based on the functional spec and domain experience.
2. Orthogonal Array Testing Strategy to generate the optimized test set using OA Tool.
OATS Implementation - A Case StudyOATS Implementation - A Case Study
13
14
OATS Implementation - A Case Study contd..OATS Implementation - A Case Study contd..
TE Effort: 9 mds TE Effort: 3 mds
STEP 1: Identify the Factors and Values for generating the Test Set.
STEP 2: Identify any dependency of values between Factors to avoid incorrect combinations ex. Floating cart is enabled only when mini-cart is enabled.
Applying the TechniqueApplying the Technique
15
16
STEP 3: Input the parameters as Factors and Values OA Tool that generates Orthogonal Array Test Set.
FIGURE1 Optimized Test Table using rdExpert Tool
Applying the TechniqueApplying the Technique
17
STEP 4: Analyze the test set generated to ensure that it covers required cases.
If not, modify the Factors and Values to generate another set. This may be done by splitting the Factors and getting 2 different sets.
Add any manual cases which are not part of the test set.
Incorporate any boundary values for parameters which are required to be tested.
Applying the TechniqueApplying the Technique
STEP 5: Generate the Test Spec in required format using the Script functionality of the Tool.
Figure2 Script for converting matrix to Test Spec
Applying the TechniqueApplying the Technique
18
ROI Analysis for ReleaseROI Analysis for Release
OATS has been used extensively in many releases and off-cycles with excellent ROI:
19
What works! and What Doesn’t!What works! and What Doesn’t! Applying OATS manually is not advisable.
Focusing the testing effort on the wrong area of the application.
Picking the wrong parameters to combine.
Orthogonal array only tests the most optimal combinations and not all.
One must use their testing skills, expertise, and experience to improve the test cases produced by these methods.
20
Execute Endlessly or Efficiently?
OROR
Execute Efficiently?
Enables to select a small test set
High functional and code cvg
Increased overall productivity
Systematic scoping of cases
Easy Analysis
Optimal implementation time
Minimal Test Execution
Simpler and less error prone
Useful for integration and
regression testing
Use OATSExecute Endlessly?
Use Conventional Approach
21
References
22
[1] Orthogonal Array Testing Strategy (OATS) Technique - Jeremy M. Harrell
[2] Orthogonally Speaking, StickyMinds. (2001):
http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=MAGAZINE&ObjectId=5031&tth=DYN&tt=siteemail&iDyn=2#table3go#table3go
[3] http://www.pairwise.org/tools.asp
[4] http://upiia.uab.es/teach/a21291/apunts/provaOO/OATS.pdf
[5] http://www.combinatorialtesting.com/clear-introductions