Risk base effective testing and quality management in the project
-
Upload
stowarzyszenie-jakosci-systemow-informatycznych-sjsi -
Category
Leadership & Management
-
view
307 -
download
0
Transcript of Risk base effective testing and quality management in the project
MOTOROLA SOLUTIONS
09 Oct 15
Risk-based Testing & Quality management in a program\project.
© 2015 Motorola Solutions, Inc.
ARTUR GORSKI
Agenda 1. Problem description.
2. Risk 1 - # of undiscovered defects.
3. Risk 2 - # of failing customer scenarios.
4. Risk 3 - # of non-detectable defects.
5. Summary.
© 2015 Motorola Solutions, Inc.
Agenda 1. Problem description.
2. Risk 1 - # of undiscovered defects.
3. Risk 2 - # of failing customer scenarios.
4. Risk 3 - # of non-detectable defects.
5. Summary.
© 2015 Motorola Solutions, Inc.
© 2015 Motorola Solutions, Inc.
IDEAL IMAGE
1. We know the number of undiscovered defects in each module of the product.
2. We know distribution of Critical, Not-Critical and Cosmetic defects of
undetected faults.
3. We know tests automation robustness for regression testing.
4. We know the number of defects that won`t be detected without e.g. SIT.
© 2015 Motorola Solutions, Inc.
Agenda 1. Problem description.
2. Risk 1 - # of undiscovered defects.
3. Risk 2 - # of failing customer scenarios.
4. Risk 3 - # of non-detectable defects.
5. Summary.
© 2015 Motorola Solutions, Inc.
Risk #1 - # of undiscovered defects
© 2015 Motorola Solutions, Inc.
Due to the fact that development introduces defects, there is a possibility of finding them in a late phases,
what can interfere with quality gates and delay release.
Let`s use historical data related to our project/product.
This is example data
© 2015 Motorola Solutions, Inc.
And pick those that had the same testing process And where estimates related to implementation tasks
were reliable and available.
This is example data
© 2015 Motorola Solutions, Inc.
Gather the data and calculate actual injection rate (# of defects injected per 1 ideal day of implementation).
Release Dev [ID] # of sourced
defects Real Inj.
Rate Improvement
Project C 500 250 0.5 Baseline
Project E 800 320 0.4 20%
Actual project 0.32 20% (planned)
This is example data
© 2015 Motorola Solutions, Inc.
Compare parameters values of a static model, which will help you to judge whether the conditions of
particular modules are better or worse.
This is example data
© 2015 Motorola Solutions, Inc.
Compare parameters values of a static model, which will help you to judge whether the conditions of
particular modules are better or worse.
This is example data
© 2015 Motorola Solutions, Inc.
Now… what are the main factors that influence number of injected defects ?
Technical complexity
Avg. injection rate
Estimate
Static model
© 2015 Motorola Solutions, Inc.
Technical Complexity Assessment From 1 to 5 (1 – the easiest)
Tester Developer Architect
3 2 4 5 4 4
© 2015 Motorola Solutions, Inc.
This is example data
Technical Complexity Assessment From 1 to 5 (1 – the easiest)
Count average difference
© 2015 Motorola Solutions, Inc.
This is example data
Average Injection Rate is taken from static model
This is example data
© 2015 Motorola Solutions, Inc.
Estimates are exported from Version1 and compared with burn down chart.
© 2015 Motorola Solutions, Inc.
This is example data
So we are having technical complexity, avg. inj. Rate and estimates.
Let`s simulate # of defects that were injected.
This is example data
© 2015 Motorola Solutions, Inc.
What is Level Of Trust ?
It`s a level of confidence in the received data. From 1 to 6 (1 - complete lack of trust)
LoT should be high (5/6)
LoT should be moderate
© 2015 Motorola Solutions, Inc.
This is example data
in a case of technical complexity
LoT should be high (5/6)
LoT should be low
Count average difference
LoT should be moderate
is moderate
© 2015 Motorola Solutions, Inc.
This is example data
in a case of avg. injection rate think about border values and count disorder
LoT = 5
LoT = 3
[(Max – Avg)/ Avg] * 100%
© 2015 Motorola Solutions, Inc.
This is example data
In practice…
The lower the value the more disorders given input AND
the higher probability of choosing disordered value during simulation.
LoT Value Disorder in %
6 0%
5 20%
4 40%
3 60%
2 80%
1 100%
© 2015 Motorola Solutions, Inc.
Example
Estimate = 10
- 60% disorder + 60% disorder
Given value
- 20% disorder + 20% disorder
LoT_1 = 3
LoT_2 = 5
© 2015 Motorola Solutions, Inc.
When the parameters are set let`s run simulation.
When the parameters are set, let`s run the simulation. # of simulations
After each simulation each total number of defects is stored.
This is example data
© 2015 Motorola Solutions, Inc.
When the parameters are set let`s run simulation.
Monte Carlo
Each result is stored
When simulation is finished a histogram of outputs is generated.
The lowest
The highest
The most probable
© 2015 Motorola Solutions, Inc.
This is example data
When the parameters are set let`s run simulation.
Then cumulative representation is constructed
Monitor and control found vs. injected defects in a module. This is example data
© 2015 Motorola Solutions, Inc.
Injection and found curves for P1\F1
When the parameters are set let`s run simulation.
I ‘know’ how many defects are left, but ‘which’ ones ?
© 2015 Motorola Solutions, Inc.
This is example data
Injection and found curves for P1\F1
When the parameters are set let`s run simulation.
Now we can assess the risk #1
This is example data
© 2015 Motorola Solutions, Inc.
Injection and found curves for P1\F1
When the parameters are set let`s run simulation.
Probability and Impact are used to assess the overall risk Summary of Risk #1 for each module:
P1/F1 P2/F2 P3/F3 Etc.
© 2015 Motorola Solutions, Inc.
When the parameters are set let`s run simulation.
What can we do with the information ?
Summary of Risk #1 for each module: P1/F1 P2/F2 P3/F3 Etc.
Prioritize execution of test cases Deliver information to key stakeholders about actual product quality Add/Remove testing effort etc.
© 2015 Motorola Solutions, Inc.
When the parameters are set let`s run simulation.
How do we check whether the prediction is correct ? Wait for feedback from SIT, FAT, UAT, Client usage etc.
Use Software Reliability Engineering e.g. Weibull distributions
CDF: F(t) = 1-e-(t/c)^m
PDF: f(t) = (2/t)(t/c)me -(t/c)^m
Metrics and Models in Software Quality Engineering, 2nd Edition [Chapter 7] Wikipedia.org
© 2015 Motorola Solutions, Inc.
When the parameters are set let`s run simulation.
How do we check whether the prediction is correct ?
SRE
This is example data
© 2015 Motorola Solutions, Inc.
Injection and found curves for P1\F1
Agenda 1. Problem description.
2. Risk 1 - # of undiscovered defects.
3. Risk 2 - # of failing customer scenarios.
4. Risk 3 - # of non-detectable defects.
5. Summary.
© 2015 Motorola Solutions, Inc.
Divide your requirements/specifications Requirements/Specifications
# of Automatable # of Non automatable
Module automatability
# of Automated # of Not Automated
Automation status of a module
# of Non automatable + # of Not Automated Risk impact
Risk probability
Risk level
© 2015 Motorola Solutions, Inc.
Example data
Automatability
Automation status
This is example data
P2/F2 Automatability
P2/F2 Automation Status P3/F3 Automation Status
P3/F3 Automatability
© 2015 Motorola Solutions, Inc.
Example data
This is example data
P2/F2 Automatability
P2/F2 Automation Status P3/F3 Automation Status
P3/F3 Automatability
© 2015 Motorola Solutions, Inc.
Risk overview Risk #2:
Due to the fact that defects can be injected into already tested areas during on-going development or during defects fixing there is a possibility of delivering a feature with failing functionalities covered by requirements what can decrease customer satisfaction. Probability: Automatability level and Automation Status e.g. 98% and 77% Impact: Number of not automated specifications e.g. 15
© 2015 Motorola Solutions, Inc.
This is example data
Use probability and impact to assess the risk Summary of Risk #2 for each module:
P2/F2 P3/F3 etc.
© 2015 Motorola Solutions, Inc.
What can we do with the information ?
Summary of Risk #2 for each module: P2/F2 P3/F3 etc.
Change automation strategy to increase automatability of a module Invest more and increase automation level Plan proper manual regression etc.
© 2015 Motorola Solutions, Inc.
Agenda 1. Problem description.
2. Risk 1 - # of undiscovered defects.
3. Risk 2 - # of failing customer scenarios.
4. Risk 3 - # of non-detectable defects.
5. Summary.
© 2015 Motorola Solutions, Inc.
Why some of defects are non-detectable during your testing phase ?
Some will be injected after your`s testing phase; Some require software-hardware integration; Some require integration with other systems; Some require clients` environment; Some require specific configuration; Some require specific perspective and higher independency; Some require real devices not simulators; etc.
© 2015 Motorola Solutions, Inc.
S
Assess defects that escaped from previous release
Around 61% were Impossible or Hard to find during our testing phase
This is example data
© 2015 Motorola Solutions, Inc.
Risk overview Risk #3:
Due to the fact that some of the defects are related to hardware, software and hardware integration, real user environment, systems integration and are impossible to be detected by simulators there is a possibility of detecting critical defects (that can mask other defects) during e.g. SIT testing phase what can interfere with quality gates and delay release of a program. Probability: Activities and its results done by other testing teams, during demos, dependency to hardware and other system etc. Impact: 61% of undiscovered defects
© 2015 Motorola Solutions, Inc.
Summary of Risk #3 for each module: P2/F2 P3/F3 etc.
Use probability and impact to assess the risk
© 2015 Motorola Solutions, Inc.
What can we do with the information ?
Summary of Risk #3 for each module: P2/F2 P3/F3 etc.
Inform stakeholders about risk Plan early system integration Built new skills in your team etc.
© 2015 Motorola Solutions, Inc.
Agenda 1. Problem description.
2. Risk 1 - # of undiscovered defects.
3. Risk 2 - # of failing customer scenarios.
4. Risk 3 - # of non-detectable defects.
5. Summary.
© 2015 Motorola Solutions, Inc.
When the parameters are set let`s run simulation.
Plan risks reactions and manage the risks.
Feature Risk #1 Risk #2 Risk #3 Risk reactions
P2/F2 accept
P3/F3
1. Invest in exploratory testing
2. Plan proper manual regression
3. Plan ESI with SIT team
P1/F1 …
This is example data
© 2015 Motorola Solutions, Inc.
© 2015 Motorola Solutions, Inc.
Any questions ? ARTUR GORSKI pl.linkedin.com/in/gorskiartur [email protected]