כיצד ניתן לצמצם את כמות הבדיקות מבלי לפגוע (ואף לשפר)...
description
Transcript of כיצד ניתן לצמצם את כמות הבדיקות מבלי לפגוע (ואף לשפר)...
IBM Haifa Research Lab
© 2010 IBM Corporation
Combinatorial Test Design and the FoCuS Tool
ItaiItai SegallSegall
Software Testing Verification and Reviews Software Testing Verification and Reviews
GroupGroup
IBM Haifa Research Lab
© 2010 IBM Corporation
About us
� Software Testing, Verification and Review Methodologies group at IBM Haifa Research Labs
– Concurrent testing
– Code review methodology and tool support
– Code coverage analysis
– Test selection
– Functional modeling and test planning
IBM Haifa Research Lab
© 2010 IBM Corporation33
Outline
� Combinatorial Test Design (CTD) Concepts and Overview
� The FoCuS Tool and the Modeling and Reduction Process
IBM Haifa Research Lab
© 2010 IBM Corporation44
Observation - Omission is a Major Risk in Testing
� Choosing what to test out of an enormous test space
– What is a test space? The set of parameters/values that define the
scenarios, inputs, configurations or conditions to be tested, such as
• Customer type (business, residential, government, …)
• Customer geography (provinces)
• Transaction type (add service, delete service, request credit …)
� When you have a list of 3000 tests, can it be reviewed?
� How do you know what’s there and what is missing?
– Coverage (in retrospect)
– Preventive techniques using CTD
IBM Haifa Research Lab
© 2010 IBM Corporation55
An Example of a Test Design
� The configuration space for a cellphone application
– Bluetooth – Yes / No
– GPS – Yes / No
– GPS software
– Support for different GSM frequencies
– Support for different 3G frequencies
– Support for EMS and MMS
� Total number of combinations
– 2 x 2 x 5 x 2 x 2 x 2 x 2 x 2 x 2 x 2 x 2 = 5120 Too many!
IBM Haifa Research Lab
© 2010 IBM Corporation
Do we really need to test all 5120 configurations?
66
� Source http://csrc.nist.gov/groups/SNS/acts/ftfi.html used in slides 6,8,9
So, to balance cost and risk, we look to select a subset of tests that covers the interactions of configuration variables at some level of interaction (pairs, three-
way, etc.)
IBM Haifa Research Lab
© 2010 IBM Corporation
How do we reduce the 5120 potential test cases?
� Interactions !
– Interaction level of every 2 means that for any two variables all the combinations of their values have been tested together.
– Every three is the same for every three variables
� IBM’s FoCuS tool models the variables, and allows us to adjust the level of interaction/coverage to achieve acceptable cost/risk tradeoff
� Explicit identification of what is tested and what is not
tested
IBM Haifa Research Lab
© 2010 IBM Corporation88
How do we reduce the 5120 potential test cases?
� The configuration space for a cellphone application
– Bluetooth – Yes / No
– GPS – Yes / No
– GPS software
– Support for 3 different GSM frequencies
– Support for 2 different 3G frequencies
– Support for EMS and MMS
� Total number of combinations
– 2 x 2 x 5 x 2 x 2 x 2 x 2 x 2 x 2 x 2 x 2 = 5120
� Question: How many tests are required for covering every combination of every 2 parameters?
� Answer: only 16 ! (99.7% reduction from 5120)
IBM Haifa Research Lab
© 2010 IBM Corporation
How do we reduce the 5120 potential test cases?
TRUETRUEFALSETRUEFALSETRUEFALSEFALSENoneFALSEFALSE
TRUETRUETRUEFALSETRUETRUEFALSEFALSEuDriveTRUETRUE
FALSEFALSEFALSETRUETRUETRUETRUETRUEuDriveTRUETRUE
TRUETRUEFALSEFALSEFALSETRUEFALSEFALSEiGoTRUEFALSE
FALSEFALSEFALSEFALSETRUETRUEFALSEFALSEuDriveTRUETRUE
FALSEFALSETRUEFALSEFALSETRUEFALSETRUENoneFALSETRUE
TRUEFALSEFALSETRUETRUETRUETRUETRUEmioMapTRUEFALSE
TRUETRUEFALSETRUEFALSEFALSEFALSEFALSEDestinatorTRUETRUE
TRUETRUETRUEFALSETRUETRUEFALSEFALSEDestinatorTRUETRUE
FALSETRUEFALSEFALSEFALSEFALSEFALSEFALSEmioMapTRUETRUE
FALSEFALSETRUETRUETRUETRUEFALSETRUEiGoTRUETRUE
TRUETRUEFALSETRUETRUEFALSETRUETRUEiGoTRUEFALSE
TRUEFALSETRUEFALSEFALSEFALSETRUETRUEmioMapTRUEFALSE
TRUEFALSEFALSETRUETRUEFALSETRUETRUENoneFALSETRUE
FALSEFALSETRUETRUEFALSEFALSETRUEFALSEuDriveTRUEFALSE
FALSEFALSETRUEFALSETRUEFALSETRUETRUEDestinatorTRUEFALSE
MMSEMSUMTS2100UMTS850GSM1800GSM900GSM850Wi_Fi
GPS_Soft
wareGPSBluetooth
Interaction level of every 2 requires 16 tests – 99.7% reduction from 5120
IBM Haifa Research Lab
© 2010 IBM Corporation
How do we reduce the 5120 potential test cases?
TRUETRUEFALSETRUEFALSETRUEFALSEFALSENoneFALSEFALSE
TRUETRUETRUEFALSETRUETRUEFALSEFALSEuDriveTRUETRUE
FALSEFALSEFALSETRUETRUETRUETRUETRUEuDriveTRUETRUE
TRUETRUEFALSEFALSEFALSETRUEFALSEFALSEiGoTRUEFALSE
FALSEFALSEFALSEFALSETRUETRUEFALSEFALSEuDriveTRUETRUE
FALSEFALSETRUEFALSEFALSETRUEFALSETRUENoneFALSETRUE
TRUEFALSEFALSETRUETRUETRUETRUETRUEmioMapTRUEFALSE
TRUETRUEFALSETRUEFALSEFALSEFALSEFALSEDestinatorTRUETRUE
TRUETRUETRUEFALSETRUETRUEFALSEFALSEDestinatorTRUETRUE
FALSETRUEFALSEFALSEFALSEFALSEFALSEFALSEmioMapTRUETRUE
FALSEFALSETRUETRUETRUETRUEFALSETRUEiGoTRUETRUE
TRUETRUEFALSETRUETRUEFALSETRUETRUEiGoTRUEFALSE
TRUEFALSETRUEFALSEFALSEFALSETRUETRUEmioMapTRUEFALSE
TRUEFALSEFALSETRUETRUEFALSETRUETRUENoneFALSETRUE
FALSEFALSETRUETRUEFALSEFALSETRUEFALSEuDriveTRUEFALSE
FALSEFALSETRUEFALSETRUEFALSETRUETRUEDestinatorTRUEFALSE
MMSEMSUMTS2100UMTS850GSM1800GSM900GSM850Wi_Fi
GPS_Soft
wareGPSBluetooth
Interaction level of every 2 requires 16 tests – 99.7% reduction from 5120
IBM Haifa Research Lab
© 2010 IBM Corporation
How do we reduce the 5120 potential test cases?
TRUETRUEFALSETRUEFALSETRUEFALSEFALSENoneFALSEFALSE
TRUETRUETRUEFALSETRUETRUEFALSEFALSEuDriveTRUETRUE
FALSEFALSEFALSETRUETRUETRUETRUETRUEuDriveTRUETRUE
TRUETRUEFALSEFALSEFALSETRUEFALSEFALSEiGoTRUEFALSE
FALSEFALSEFALSEFALSETRUETRUEFALSEFALSEuDriveTRUETRUE
FALSEFALSETRUEFALSEFALSETRUEFALSETRUENoneFALSETRUE
TRUEFALSEFALSETRUETRUETRUETRUETRUEmioMapTRUEFALSE
TRUETRUEFALSETRUEFALSEFALSEFALSEFALSEDestinatorTRUETRUE
TRUETRUETRUEFALSETRUETRUEFALSEFALSEDestinatorTRUETRUE
FALSETRUEFALSEFALSEFALSEFALSEFALSEFALSEmioMapTRUETRUE
FALSEFALSETRUETRUETRUETRUEFALSETRUEiGoTRUETRUE
TRUETRUEFALSETRUETRUEFALSETRUETRUEiGoTRUEFALSE
TRUEFALSETRUEFALSEFALSEFALSETRUETRUEmioMapTRUEFALSE
TRUEFALSEFALSETRUETRUEFALSETRUETRUENoneFALSETRUE
FALSEFALSETRUETRUEFALSEFALSETRUEFALSEuDriveTRUEFALSE
FALSEFALSETRUEFALSETRUEFALSETRUETRUEDestinatorTRUEFALSE
MMSEMSUMTS2100UMTS850GSM1800GSM900GSM850Wi_Fi
GPS_Soft
wareGPSBluetooth
Interaction level of every 2 requires 16 tests – 99.7% reduction from 5120
IBM Haifa Research Lab
© 2010 IBM Corporation
How do we reduce the 5120 potential test cases?
TRUETRUEFALSETRUEFALSETRUEFALSEFALSENoneFALSEFALSE
TRUETRUETRUEFALSETRUETRUEFALSEFALSEuDriveTRUETRUE
FALSEFALSEFALSETRUETRUETRUETRUETRUEuDriveTRUETRUE
TRUETRUEFALSEFALSEFALSETRUEFALSEFALSEiGoTRUEFALSE
FALSEFALSEFALSEFALSETRUETRUEFALSEFALSEuDriveTRUETRUE
FALSEFALSETRUEFALSEFALSETRUEFALSETRUENoneFALSETRUE
TRUEFALSEFALSETRUETRUETRUETRUETRUEmioMapTRUEFALSE
TRUETRUEFALSETRUEFALSEFALSEFALSEFALSEDestinatorTRUETRUE
TRUETRUETRUEFALSETRUETRUEFALSEFALSEDestinatorTRUETRUE
FALSETRUEFALSEFALSEFALSEFALSEFALSEFALSEmioMapTRUETRUE
FALSEFALSETRUETRUETRUETRUEFALSETRUEiGoTRUETRUE
TRUETRUEFALSETRUETRUEFALSETRUETRUEiGoTRUEFALSE
TRUEFALSETRUEFALSEFALSEFALSETRUETRUEmioMapTRUEFALSE
TRUEFALSEFALSETRUETRUEFALSETRUETRUENoneFALSETRUE
FALSEFALSETRUETRUEFALSEFALSETRUEFALSEuDriveTRUEFALSE
FALSEFALSETRUEFALSETRUEFALSETRUETRUEDestinatorTRUEFALSE
MMSEMSUMTS2100UMTS850GSM1800GSM900GSM850Wi_Fi
GPS_Soft
wareGPSBluetooth
Interaction level of every 2 requires 16 tests – 99.7% reduction from 5120
IBM Haifa Research Lab
© 2010 IBM Corporation1313
Outline
� Combinatorial Test Design (CTD) Concepts and Overview
� The FoCuS Tool and The Modeling and Reduction Process
IBM Haifa Research Lab
© 2010 IBM Corporation1414
A Comment on the Use of CTD in the Industry
� Many CTD tools were created
– The mathematics are very nice to play with
– The problem is solved with advanced combinatorics
� However, most are almost toy-tools
– Real world problems are not toy examples
– Real world problems may be hard to model
– Not all value combinations are possible – there are restrictions
– Existing tests need to be evaluated or taken into account
– Advanced requirements such as distribution requirements on the values in the resulting tests
� Most tools miss many things testers need
IBM Haifa Research Lab
© 2010 IBM Corporation
CTD within our FoCuS tool
� Tackles complexity of modeling by Cartesian product methodology
� Fully supports restrictions on the valid combinations
� Highly scalable
� Supports advanced features
– Distribution requirements
– Good path and bad path test planning
� Considers existing tests
– Report uncovered areas
– Suggest complementary test plans
15
IBM Haifa Research Lab
© 2010 IBM Corporation
The cell phone example in FoCuS
IBM Haifa Research Lab
© 2010 IBM Corporation
Example – cont.
� Cartesian product consists of 5120 possible configurations:
IBM Haifa Research Lab
© 2010 IBM Corporation
Example – Cont.
� Some combinations do not make sense, and should be removed:
IBM Haifa Research Lab
© 2010 IBM Corporation
Example – Cont.
� Now the model contains 2560 legal tasks:
IBM Haifa Research Lab
© 2010 IBM Corporation
Example – Cont.
� A test plan can now be created:
IBM Haifa Research Lab
© 2010 IBM Corporation
Example – Cont.
� 16 tests suffice:
IBM Haifa Research Lab
© 2010 IBM Corporation
Example – Cont.
� Suppose I have only a few Bluetooth components
IBM Haifa Research Lab
© 2010 IBM Corporation
Example – Cont.
� The new test plan:
IBM Haifa Research Lab
© 2010 IBM Corporation
Example – Cont.
� More complex coverage requirements may be stated:
IBM Haifa Research Lab
© 2010 IBM Corporation
Example – cont.
� And the resulting test plan:
IBM Haifa Research Lab
© 2010 IBM Corporation
Example – cont.
� “.. But I already have some tests implemented, I want to use them”
IBM Haifa Research Lab
© 2010 IBM Corporation
Example – cont.
� FoCuS can analyze what is not covered:
IBM Haifa Research Lab
© 2010 IBM Corporation
Example – cont.
IBM Haifa Research Lab
© 2010 IBM Corporation
Example – cont.
� .. and suggest a complementary test plan:
IBM Haifa Research Lab
© 2010 IBM Corporation3030
Comparing Manually Created Test Plans to Usage of Modeling and CTD
�Manual test plans will usually have many large holes
– And it is not known where they are
�Manual test plans are more expensive
– Take a lot of time to design
– Manual test plans contain more tests, so longer time to implement the tests and longer time to execute them
�Known quality situation
– With CTD you can get the desired quality based on risk
– With CTD you can compare alternatives, as once the model is created, creating alternatives takes minutes
IBM Haifa Research Lab
© 2010 IBM Corporation31
Additional Advantages of Systematic Test Plan Design
� Once the methodology is learned, it is much less work – less tests to execute
� The tool tells you how many test configurations are needed and specific configurations that fulfill your requirements
– You can discuss alternatives for how the budget will be used
� Having a model instead of a spreadsheet test list is much more useful
– Less effort to maintain the model
– From the model you can generate lists after changes are made
• A new operating system was added
• Need to add a new variable that interacts with the current ones
• Need to remove variables or values and make sure that the test plan maintains quality
– The model is much easier to review than traditional test plans as the
generation scheme explicitly describes what is tested
IBM Haifa Research Lab
© 2010 IBM Corporation
Thank You
� Trial download: http://www.alphaworks.ibm.com/tech/focus
� Contact me at: [email protected]