Matthias Ratert - Automated Test Case Prioritization - EuroSTAR 2012
-
Upload
eurostar-software-testing-conference -
Category
Technology
-
view
145 -
download
0
description
Transcript of Matthias Ratert - Automated Test Case Prioritization - EuroSTAR 2012
![Page 1: Matthias Ratert - Automated Test Case Prioritization - EuroSTAR 2012](https://reader033.fdocuments.net/reader033/viewer/2022052621/55893201d8b42add138b458d/html5/thumbnails/1.jpg)
Matthias Ratert, Secusmart GmbH
Automated, tool independent
test case prioritization
www.eurostarconferences.com
@esconfs #esconfs
![Page 2: Matthias Ratert - Automated Test Case Prioritization - EuroSTAR 2012](https://reader033.fdocuments.net/reader033/viewer/2022052621/55893201d8b42add138b458d/html5/thumbnails/2.jpg)
• Secusmart GmbH (Germany)
– Secure voice, messaging & email (NATO Restricted)
– Hardware based encryption and authentication
in mobile networks as well as landline networks
– Convenient, secure and with crystal-clear voice quality
– Mobile clients for Nokia, Blackberry, Android and iPhone
• Matthias Ratert
– 14 years experience in Software development & testing
– Worked within automotive and mobile phone industry
Introduction
![Page 3: Matthias Ratert - Automated Test Case Prioritization - EuroSTAR 2012](https://reader033.fdocuments.net/reader033/viewer/2022052621/55893201d8b42add138b458d/html5/thumbnails/3.jpg)
• Many cases
– Short test cases
– Hierarchical and linked
– Just a few expected results
• A lot cases
– Just headlines
– No expected results
What do you prefer?
Test case
• A few (long?) cases
– Detailed test steps
– A lot of expected results
• More cases
– Adding more tests
– Splitting tests
![Page 4: Matthias Ratert - Automated Test Case Prioritization - EuroSTAR 2012](https://reader033.fdocuments.net/reader033/viewer/2022052621/55893201d8b42add138b458d/html5/thumbnails/4.jpg)
• Are you afraid to expand your test suite continuously?
– Will there be time to execute all of them in the future?
– Will I spend too much time to create & maintain the suite?
– Will I be able to select the “important” test cases?
I was !
Test case
![Page 5: Matthias Ratert - Automated Test Case Prioritization - EuroSTAR 2012](https://reader033.fdocuments.net/reader033/viewer/2022052621/55893201d8b42add138b458d/html5/thumbnails/5.jpg)
• 1st prioritization: Necessity – Is this test case needed?
– Define the test case or not?
– Everyone is doing this...
• 2nd prioritization: Importance – Planned execution & code/requirements coverage:
• Regression tests (high importance)
• Execute often (medium importance)
• Execute seldom (low importance)
– Normally part of the test case definition
– Many Test Management tools are supporting this !!!
1st & 2nd test case prioritization
![Page 6: Matthias Ratert - Automated Test Case Prioritization - EuroSTAR 2012](https://reader033.fdocuments.net/reader033/viewer/2022052621/55893201d8b42add138b458d/html5/thumbnails/6.jpg)
• Test session planning with one or several input factors:
– New functionality implemented
– Functionality changed (e.g. CRs)
– Bugs fixed
– Risks (error status/statistics, important/critical areas)
– Past test execution coverage (Tested before? How often?)
– Past test result ratio (e.g. failed before)
– Error guessing techniques
– Internal milestone
– Important deliverable
– External (customer) deliverable
3rd prioritization: Urgency
![Page 7: Matthias Ratert - Automated Test Case Prioritization - EuroSTAR 2012](https://reader033.fdocuments.net/reader033/viewer/2022052621/55893201d8b42add138b458d/html5/thumbnails/7.jpg)
• Done by the Test Manager before & during the test session – Ensure an optimal test coverage
– Maximize the rate of fault detection
– Get more time for Exploratory and Non-Functional Testing
• No good tool support (AFAIK) MANUAL WORK
• Time consuming – Thorough prioritization decisions need to be made
• Complex & error prone
– Applying different strategies is sometimes even impossible
3rd prioritization: Urgency
Can we support the Test Manager?
![Page 8: Matthias Ratert - Automated Test Case Prioritization - EuroSTAR 2012](https://reader033.fdocuments.net/reader033/viewer/2022052621/55893201d8b42add138b458d/html5/thumbnails/8.jpg)
• Easy to automate:
– Past test execution coverage (Tested before? How often?)
– Past test result ratio (e.g. failed before)
Started with in the 1st thesis
• Good to automate:
– Bugs fixed
– Risks (error status/statistics incl. error severity)
Covered with the 2nd thesis
Automated test case prioritization
![Page 9: Matthias Ratert - Automated Test Case Prioritization - EuroSTAR 2012](https://reader033.fdocuments.net/reader033/viewer/2022052621/55893201d8b42add138b458d/html5/thumbnails/9.jpg)
• Possible to automate: – New functionality – Functionality changed – Risks (important/critical areas) – Important deliverable
Provide the possibility to automate later
• Hard to automate: – Error guessing techniques – Internal milestone – External (customer) deliverable
Do not automated
Automated test case prioritization
![Page 10: Matthias Ratert - Automated Test Case Prioritization - EuroSTAR 2012](https://reader033.fdocuments.net/reader033/viewer/2022052621/55893201d8b42add138b458d/html5/thumbnails/10.jpg)
• 1st thesis:
Implement prioritization algorithm in TestLink
– Only test execution data from TestLink is used
– Internal databases can be used / “tweaked” easily
– Provide a priority filter for the Tester during test execution
• 2nd thesis:
Implement tool independent, stand-alone prioritization tool
– All kind of data (e.g. error DB, requirements tool) are gathered
– Each test management tool can be “tweaked”
The 2 diploma theses
![Page 11: Matthias Ratert - Automated Test Case Prioritization - EuroSTAR 2012](https://reader033.fdocuments.net/reader033/viewer/2022052621/55893201d8b42add138b458d/html5/thumbnails/11.jpg)
1st thesis: The basic idea
Test Case 3 Session n-1 Session n-2 Session n-3 Session n-4
Test Case 1 Session n-1 Session n-2 Session n-3 Session n-4
Passed Priority ---- Priority --- Priority -- Priority -
Failed Priority + Priority ++ Priority +++ Priority ++++
Not executed Priority +++++
Test Case 2 Session n-1 Session n-2 Session n-3 Session n-4
Passed Priority ---- Priority --- Priority -- Priority -
Failed Priority + Priority ++ Priority +++ Priority ++++
Not executed Priority +++++
…
![Page 12: Matthias Ratert - Automated Test Case Prioritization - EuroSTAR 2012](https://reader033.fdocuments.net/reader033/viewer/2022052621/55893201d8b42add138b458d/html5/thumbnails/12.jpg)
2 approaches to provide the priority filter to the Tester:
1. The build-in TestLink prioritization
+ Minimal changes to TestLink
- Only 3 filter level
- Each priority level is selected exclusively,
higher priority levels are not included in the filter
1st thesis: Test execution filter
![Page 13: Matthias Ratert - Automated Test Case Prioritization - EuroSTAR 2012](https://reader033.fdocuments.net/reader033/viewer/2022052621/55893201d8b42add138b458d/html5/thumbnails/13.jpg)
The build-in TestLink prioritization
• The urgency is set manually
• Only 3 priorities available (High, Medium, Low), not expandable
High (3) Medium (2) Low (1)
High (3)
Medium (2)
Low (1)
Importance
Urgency
High (9)
High (6)
High (6)
Medium (4)
Priority = Test Case importance x urgency
Medium (3) Low (2)
Medium (3)
Low (2)
Low(1)
![Page 14: Matthias Ratert - Automated Test Case Prioritization - EuroSTAR 2012](https://reader033.fdocuments.net/reader033/viewer/2022052621/55893201d8b42add138b458d/html5/thumbnails/14.jpg)
The build-in TestLink prioritization
2. Only cases with the selected
priority are shown
1. Tester uses the TestLink filter to
find the cases to be executed
![Page 15: Matthias Ratert - Automated Test Case Prioritization - EuroSTAR 2012](https://reader033.fdocuments.net/reader033/viewer/2022052621/55893201d8b42add138b458d/html5/thumbnails/15.jpg)
• Now the urgency is set automatically at the beginning of a test session
• The whole algorithm and the priority classification are configurable
• Recommendation is to set:
The modified TestLink prioritization
High (10) Medium (4) Low (1)
High (3)
Medium (2)
Low (1)
Importance
Urgency
High (30)
High (20)
High (12)
Medium (8)
Priority = Test Case importance x urgency
Low (2)
Low(1) High (10) Medium (4)
Low (3)
– 10% - 20% test cases to HIGH
– 20% - 50% test cases to MEDIUM
![Page 16: Matthias Ratert - Automated Test Case Prioritization - EuroSTAR 2012](https://reader033.fdocuments.net/reader033/viewer/2022052621/55893201d8b42add138b458d/html5/thumbnails/16.jpg)
2 approaches to provide the priority filter to the Tester: 1. The build-in TestLink prioritization
+ Minimal changes to TestLink
- Only 3 filter level
- Each priority level is selected exclusively
2. A new filter in the test execution
+ As many filter level as wanted (chosen: 15)
good granularity
+ Each priority level includes all higher priorities
+ Manual urgency can still be used
- A lot of changes to TestLink sources
1st thesis: Test execution filter
![Page 17: Matthias Ratert - Automated Test Case Prioritization - EuroSTAR 2012](https://reader033.fdocuments.net/reader033/viewer/2022052621/55893201d8b42add138b458d/html5/thumbnails/17.jpg)
TestLink with the new filter
![Page 18: Matthias Ratert - Automated Test Case Prioritization - EuroSTAR 2012](https://reader033.fdocuments.net/reader033/viewer/2022052621/55893201d8b42add138b458d/html5/thumbnails/18.jpg)
Bug severity:
Showstopper: Huge priority increase
Major: Average priority increase
Minor: Minimal priority increase
Bug status:
Fixed / recently closed: Huge priority increase
Open: Average priority increase
Closed (a while ago): Minimal priority increase
2nd thesis: Error DB involvement
Test Case 1 Session n-1 Session n-2 Session n-3 Session n-4
Passed Priority ---- Priority --- Priority -- Priority -
Failed Priority change depends on the linked bug
Not executed Priority +++++
![Page 19: Matthias Ratert - Automated Test Case Prioritization - EuroSTAR 2012](https://reader033.fdocuments.net/reader033/viewer/2022052621/55893201d8b42add138b458d/html5/thumbnails/19.jpg)
2nd thesis: Tool independency
any other data
Test Mgmt
Stand-alone tool with
configuration dashboard:
• Which algorithm(s) to prefer?
• How many test cases in highest priority?
• Visualization of the priority distribution
• …
Requirements Mgmt
Error Mgmt 1. Use interface & filter from 1st thesis 2. Use available (open) APIs
![Page 20: Matthias Ratert - Automated Test Case Prioritization - EuroSTAR 2012](https://reader033.fdocuments.net/reader033/viewer/2022052621/55893201d8b42add138b458d/html5/thumbnails/20.jpg)
• Thesis just started • Goals:
– Support different Test Management tools,
at least TestLink and HP QualityCenter
– Provide a configuration dashboard and metrics
• Select different strategies
– concentrate more (or fully) on error verification
– find as many new bugs as possible
– do mainly regression testing
– …
– Modular & generic design to enable further interfaces
and extension
2nd thesis: Tool independency
![Page 21: Matthias Ratert - Automated Test Case Prioritization - EuroSTAR 2012](https://reader033.fdocuments.net/reader033/viewer/2022052621/55893201d8b42add138b458d/html5/thumbnails/21.jpg)
By using automated test prioritization
• a huge and expanding test suite is not a risk anymore
• thorough test prioritization is ensured for every test session
• the test manager gets support especially in stress situations
• the test coverage is maximized over several releases
– time-sliced test coverage
• even low severity tests will be prioritized at some point of time
• more bugs are found
Conclusion
![Page 22: Matthias Ratert - Automated Test Case Prioritization - EuroSTAR 2012](https://reader033.fdocuments.net/reader033/viewer/2022052621/55893201d8b42add138b458d/html5/thumbnails/22.jpg)
In case if any comments and/or questions feel free to
contact me via:
Matthias Ratert
Secusmart GmbH
THANK YOU
or