-Monica Kothari

22
-Monica -Monica Kothari Kothari QE Efforts Optimization using O O rthogonal rthogonal A A rray rray T T esting esting S S trategy trategy -A Case Study 1

description

QE Efforts Optimization using O rthogonal A rray T esting S trategy - A Case Study. -Monica Kothari. Execute Smartly?. OR. Execute Endlessly?. Major Challenges. Scope. What If. Less no. of tests cases but still high functional and code coverage. - PowerPoint PPT Presentation

Transcript of -Monica Kothari

Page 1: -Monica Kothari

-Monica Kothari-Monica Kothari

QE Efforts Optimization using

OOrthogonalrthogonal AArrayrray TTestingesting SStrategytrategy

-A Case Study

1

Page 2: -Monica Kothari

Execute Smartly?

2

OR

Execute Endlessly?

Page 3: -Monica Kothari

Major Challenges

3

Scope

Page 4: -Monica Kothari

What If...

4

60% Less Effort !

• Less no. of tests cases but still high functional and code coverage

• Systematic scoping of test cases across the domain with minimal manual effort

• Find new bugs which could not be found using conventional approach

• Find bugs early in the cycle

Page 5: -Monica Kothari

Agenda

5

Complexity of the testing problem Geometric View of Test Cases Taguchi’s OATS approach Example on Orthogonal Array Why use OATS Comparison of Test Planning Methods OATS implementation Applying the Technique ROI Analysis What works and What Doesn’t!

Page 6: -Monica Kothari

Complexity of a Testing ProblemComplexity of a Testing Problem

Apply Coupon(8) Verify

Total (26)SubTotal(13)Price (5)Qty (5)Ship Tax (9)Discount (5)

Regular Item (8)Store Tag Item (8)Cross-sell Item (8)Mono-Inscn Item (8)Volmn Item (8)Incr-Decr Price (8)Softgood (8)GC (8)

Full FC 15s (16)*

Full FC (16)

Temp FC (16)

Mini-cart (16)

Verify

Buyer

Apply Coupon(8) Verify

Total (26)SubTotal(13)Price (5)Qty (5)Ship Tax (9)Discount (5)

Regular Item (8)Store Tag Item (8)Cross-sell Item (8)Mono-Inscn Item (8)Volmn Item (8)Incr-Decr Price (8)Softgood (8)GC (8)

Full FC 15s (16)*

Full FC (16)

Temp FC (16)

Mini-cart (16)

Verify

Buyer

Figure3 Test Case Coverage Flow Diagram

6

Page 7: -Monica Kothari

Key LimitationsKey Limitations

• Generating all combinations manually is error prone:

Total Combinations = Template(5)*FCType(4)*Item(8)*Action(8)*6(Verify)

= 7,680 combinations

7

• Identifying optimal combinations to test without missing any complex combinations ensuring high functional and code coverage ensuring the confidence in quality of product.

• Minimum testing cycles for execution and automation development.

Page 8: -Monica Kothari

Geometric View of Test CasesGeometric View of Test Cases

8

Page 9: -Monica Kothari

Orthogonal Array Testing StrategyOrthogonal Array Testing Strategy

Orthogonal Array is an array of values in which:

Each column is an independent variable to be tested for interaction represented as – “Factor”.

Each Factor can take a certain set of values called “Levels”.

Each row represents a test case/combination.

Taguchi OATS ApproachTaguchi OATS Approach

Orthogonal Array Testing Strategy created by Dr. Genichi Taguchi is a proven, systematic, statistical way of testing pair-wise interactions.

9

Page 10: -Monica Kothari

OATS - ExampleOATS - Example A B C

1 1 1 3

2 1 2 2

3 1 3 1

4 2 1 2

5 2 2 1

6 2 3 3

7 3 1 1

8 3 2 3

9 3 3 2

TABLE1 Sample Array using OA

Total cases = 9 which cover all pair-wise combinations of the 3 variables.

Applying OATS3 Parameters – A,B,C

3 Values – 1,2,3

All possible cases involving 3 parameters:

3*3*3 = 27 cases

10

Page 11: -Monica Kothari

Identify a minimum set of tests that will find all multi-mode defects (particularly serious defects)

• All single-mode defect will be found if every option is tested at least once (unit elements)

• 2-way or “Pair-wise” is combinations of 2 items (parameters) that cause a defect

• 3-way or “Tri-wise” is combinations of 3 items (parameters) that cause a defect

“ Pair-wise” defect detection (pair-wise testing) finds most defects, ex. testing all pairs typically

finds 75% of defects Source: Kuhn

NASA Deep Space Mission - study showed that 88% of bugs discovered using “pair-wise” defect detection testing

U.S. Food and Drug Administration – study showed 98% bugs discovered using “ pair-wise ” defect detection Source: 27 th NASA/ IEEE Software Engineering Workshop, NASA Goddard Space Flight Center, 4-6 Dec, 2002.

“ The Combinatorial Design Approach to Automatic Test Generation” Source: IEEE, Software 1996, Vol. 13, No. 5

Baseline application under study resulted in a reduction in test plan time from 1 month to less than 1 week

In several experiments, “pair-wise” demonstrated good code coverage and defect detection ability

Why OOrthogonal rthogonal AArray rray TTesting esting SStrategy?trategy?

11

Page 12: -Monica Kothari

Comparison of Test Planning MethodsComparison of Test Planning MethodsTesting Method Testing

Effort (No. Of Tests)

Testing Efforts (TestPlanning)

Ability to Find Faults

Requirements Traceability (One Factor at a Time)

Small Small Limited (a)

Exhaustive Testing Very Large Small All Faults Found (100%)

Deductive/Analytical(White-Box Testing)

Medium Very Large Close to 100%

Random/Intuitive(Based on Past

History)

Medium Medium Unpredictable

Orthogonal Array-

Based Robust Testing

Small Small Better than (a); Close to 100% when used in conjunction with code coverage analysis tools.

12

Page 13: -Monica Kothari

Both the approaches were followed and analyzed:

1. Traditional approach to generate the test spec based on the functional spec and domain experience.

2. Orthogonal Array Testing Strategy to generate the optimized test set using OA Tool.

OATS Implementation - A Case StudyOATS Implementation - A Case Study

13

Page 14: -Monica Kothari

14

OATS Implementation - A Case Study contd..OATS Implementation - A Case Study contd..

TE Effort: 9 mds TE Effort: 3 mds

Page 15: -Monica Kothari

STEP 1: Identify the Factors and Values for generating the Test Set.

STEP 2: Identify any dependency of values between Factors to avoid incorrect combinations ex. Floating cart is enabled only when mini-cart is enabled.

Applying the TechniqueApplying the Technique

15

Page 16: -Monica Kothari

16

STEP 3: Input the parameters as Factors and Values OA Tool that generates Orthogonal Array Test Set.

FIGURE1 Optimized Test Table using rdExpert Tool

Applying the TechniqueApplying the Technique

Page 17: -Monica Kothari

17

STEP 4: Analyze the test set generated to ensure that it covers required cases.

If not, modify the Factors and Values to generate another set. This may be done by splitting the Factors and getting 2 different sets.

Add any manual cases which are not part of the test set.

Incorporate any boundary values for parameters which are required to be tested.

Applying the TechniqueApplying the Technique

Page 18: -Monica Kothari

STEP 5: Generate the Test Spec in required format using the Script functionality of the Tool.

Figure2 Script for converting matrix to Test Spec

Applying the TechniqueApplying the Technique

18

Page 19: -Monica Kothari

ROI Analysis for ReleaseROI Analysis for Release

OATS has been used extensively in many releases and off-cycles with excellent ROI:

19

Page 20: -Monica Kothari

What works! and What Doesn’t!What works! and What Doesn’t! Applying OATS manually is not advisable.

Focusing the testing effort on the wrong area of the application.

Picking the wrong parameters to combine.

Orthogonal array only tests the most optimal combinations and not all.

One must use their testing skills, expertise, and experience to improve the test cases produced by these methods.

20

Page 21: -Monica Kothari

Execute Endlessly or Efficiently?

OROR

Execute Efficiently?

Enables to select a small test set

High functional and code cvg

Increased overall productivity

Systematic scoping of cases

Easy Analysis

Optimal implementation time

Minimal Test Execution

Simpler and less error prone

Useful for integration and

regression testing

Use OATSExecute Endlessly?

Use Conventional Approach

21

Page 22: -Monica Kothari

References

22

[1] Orthogonal Array Testing Strategy (OATS) Technique - Jeremy M. Harrell

[2] Orthogonally Speaking, StickyMinds. (2001):

http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=MAGAZINE&ObjectId=5031&tth=DYN&tt=siteemail&iDyn=2#table3go#table3go

[3] http://www.pairwise.org/tools.asp

[4] http://upiia.uab.es/teach/a21291/apunts/provaOO/OATS.pdf

[5] http://www.combinatorialtesting.com/clear-introductions