AmiBug.Com, Inc. August 30, 2015© Robert Sabourin, 2009Slide 1 Testing Under Pressure: Five Key...
-
Upload
charlene-bates -
Category
Documents
-
view
220 -
download
0
Transcript of AmiBug.Com, Inc. August 30, 2015© Robert Sabourin, 2009Slide 1 Testing Under Pressure: Five Key...
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2009 Slide 1
Testing Under Pressure: Five Key Principles
Robert Sabourin
President
AmiBug.Com, Inc.
Montreal, [email protected]
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2009 Slide 2
Under Pressure
• Pain points?– What hurts?– How much?
AmiBug.Com, Inc.
Testing Under Pressure
1. Begin with the end in mind
2. Active context listening
3. Decision making
4. Ruthlessly triage
5. The last best build
April 19, 2023 © Robert Sabourin, 2009 Slide 3
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2009 Slide 4
Under Pressure
(1)Begin with the end in mind
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2009 Slide 5
Fundamental Question
• How do you know when you are finished?
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2009 Slide 6
Crosby on Quality
• “Quality is defined as conformance to requirements”
• “Quality is not a measure of GOODNESS”– Phil B. Crosby, Quality is Free
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2009 Slide 7
Gerald M. Weinberg
“Quality is value to some person”
Exploring Requirements Quality
Before Design Dorset House
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2009 Slide 8
Edsger W. Dijkstra
• “Program testing can be used to show the presence of bugs, but never to show their absence”
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 9
Pareto Principle
• Vilfredo Pareto, 1848 - 1923, Economist– 80% of the wealth was in the hands of 20%
of the population
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 10
Pareto Principle
• Joseph Juran, 1903 - present,Quality Control Engineer– 1950 Quality Control Handbook
– 20% of the study population accounts for 80% of the measure under consideration
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2009 Slide 11
Under Pressure
(2) Active context listening
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2009 Slide 12
Context Drivers - BTO
• Business– Value– To whom?– Why?
• Technology– Solutions
• Organization– Corporate Structure– Team Structure– Roles and Responsibilities
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2009 Slide 14
Context Listeners
• Find Sources• Monitor Drivers• Anticipate Change• React
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 16
First Things First
• Begin with the end in mind– Gain Consensus
• Goals– How do we know we are finished?
• Purpose– Why are we doing this project?
– How will be react to change?
• Meaning– What is a bug?
– What is a test?
– What is quality?
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2009 Slide 17
"No! Try not, Do. Or do not.
There is no try."
Yoda
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 18
Bug Flow
• We will be testing … imagine that we actually find a bug!– What are we going to do about it?– How will we decide?– When should we decide how to decide?– When should we change how we decide?– When should we review our past
decisions?
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 19
Bug Flow
Entered Reviewed
Fixed
AssignedPrioritized
ClosedUnassigned
TRIAGE
REFUSE
CHECK
MANDATE
FAILURE
DESIGNATE
CORRECT
CONFIRM
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 20
Bug Workflow
Seven Steps1. Identify key stakeholders
2. Learn about decisions
3. Define bug priority & severity
4. List steps when bug is found
5. Build bug flow state model
6. Get stakeholders “buy in”
7. Adapt bug flow as required
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 21
Bug Workflow
Identify key stakeholders
1
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 22
Bug Workflow
How is a decision made?
2
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 23
Bug Workflow
UrgentSevere
UrgentNot Severe
Not UrgentSevere
Not UrgentNot Severe
Define Priority and Severity Scheme
3
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 24
Bug Workflow
Define steps to follow– bug is discovered in testing or reported from the field– a bug report form is completed– the bug report form is reviewed– the bug report is added to the bug list– a decision is made, at a bug review meeting, about whether
the bug should be fixed– if the bug is fixed then the software is re-tested to reconfirm
that the bug has indeed been fixed– if the bug is not fixed (on purpose!) then a description of the
work around is published or made available to help desk staff4
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 25
Bug Workflow
Entered Reviewed
Fixed
AssignedPrioritized
ClosedUnassigned
TRIAGE
REFUSE
CHECK
MANDATE
FAILURE
DESIGNATE
CORRECT
CONFIRM
Build State Model
5
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 26
Bug Workflow
Get stakeholder “buy-in”
6
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2009 Slide 27
R2-D2 and Chewbacca are playing the holographic game aboard the Millennium Falcon Chewbacca: Aaaaaaaaaaaaaaaarrrgh C-3PO: He made a fair move. Screaming about it can't help you.Han Solo: Let him have it. It's not wise to upset a Wookiee.C-3PO: But sir, nobody worries about upsetting a droid.Han Solo: That's 'cause droids don't pull people's arms out of their sockets when they lose. Wookiees are known to do that.Chewbacca: Grrf.C-3PO: I see your point, sir. I suggest a new strategy, R2: let the Wookiee win.
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2009 Slide 29
Under Pressure
(4) Ruthlessly triage
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 30
Test Idea Sources
• Capabilities• Failure Modes• Quality Factors• Usage Scenarios• Creative Ideas• States• Data• Environments• White Box• Taxonomies
Cap
ture
tes
ting
idea
s
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2009 Slide 31
Triage
• Criticality• Resources• Trade offs
– Benefit– Consequence– Credibility
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 32
Which test?
•Impact estimation–For each test idea guesstimate:
• benefit of implementation• consequence of implementation• benefit for not implementing• consequence of not implementing
–How credible is the information?
Tria
ge t
estin
g id
eas
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 33
How to Decide?Understanding Complex Technology Quantitatively By Tom Gilb
Rank Credibility0.0 Wild guess, no credibility
0.1 We know it has been done somewhere
0.2 We have one measurement somewhere
0.3 There are several measurements in the estimated range
0.4 The measurements are relevant to our case
0.5 The method of measurement is considered reliable
0.6 We have used the method in-house
0.7 We have reliable measurements in-house
0.8 Reliable in-house measurements correlate to independent external measurements
0.9 We have used the idea on this project and measured it
1.0 Perfect credibility, we have rock solid, contract- guaranteed, long-term, credible experience with this idea on this project and, the results are unlikely to disappear
Tria
ge t
estin
g id
eas
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 34
Which test?
Test Idea Rejection – What If?–If the cost/benefit does not make business sense then consider implementing:• part of the test, could that lead to part of the
benefit at a more reasonable cost?• more than the stated test, would that generate
more benefit?• a different test than the stated idea, could that
generate more benefit for less cost?
Tria
ge t
estin
g id
eas
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 35
Test Triage
• Test Triage– JIT Projects– High Frequency– Daily Test Triage Session– Experience dictates
• Early AM (Rob Preference)• Late PM (several clients)
Tria
ge t
estin
g id
eas
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 36
Test Triage
• Test Triage Meeting– Review Context
• Business• Technical
– Information since last triage
• Test results• Bug results• New testing ideas
Tria
ge t
estin
g id
eas
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 37
Test Triage
• Allocate Testing Assignments to Testers– Make sure testers know context– Best thing to test– Best person to test it– Best people to explore it– Best lead– Are subject matter experts required
Tria
ge t
estin
g id
eas
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 38
Test Triage
• Requirement Triage• Change Control• Test Triage• Bug Flow
– Combined– Equivalent to CCB– Few people– Fluid
Tria
ge t
estin
g id
eas
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 39
Test Triage
Life of a test ideaa. Comes into existence
b. Clarified
c. Prioritizeda. Test Now (before further testing)
b. Test before shipping
c. Nice to have
d. May be of interest in some future release
e. Not of interest in current form
f. Will never be of interest
d. Integrate into a testing objective
Tria
ge t
estin
g id
eas
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 40
Which test is next?
• Questions– Given state of project, state of business, state of
technology, our abilities, our experience and our history, what we know and what we do not know, what should we test next?
– How much effort are we willing to spend continuing to test this project?
– Can we ship yet?
Tria
ge t
estin
g id
eas
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 41
Which test is next?
• Magic crystal ball– If it existed how would you use it?– What question would you ask?– What question would it ask?
Tria
ge t
estin
g id
eas
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 42
Deciding what not to test?
• Time pressure
– Should we skip a test?– If test failed could system still be of
value to some stakeholder?– If test was skipped could important
bugs have been otherwise found?
Tria
ge t
estin
g id
eas
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 43
Guidelines and Decisions
• To each stakeholder– risk of failure– consequence of failure– value of success– how much certainty do we have – is it a wild guess or an absolute
truth?
Get
Sta
rted
Rig
ht
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2009 Slide 44
Under Pressure
(5) The last best build
AmiBug.Com, Inc.
Under Pressure
• Always know the last best build.
• Rank recent builds.• If forced to ship at a
fixed date then I often let project stakeholders trade off between recent builds.
April 19, 2023 © Robert Sabourin, 2009 Slide 45
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 47
Testing Under Pressure
So what exactly did they throw over the wall?
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 48
Getting Things Done
Adapt to change
Triage Testing
Prioritize Bugs
Track Progress
Revised risks?New test objectives?
What to test?What not to test?
What to fix?What not to fix?
What do we know so far?What don’t we know yet?
Per
iod
ic
Smoke Test Should the new build be tested at all?On failure continue with previous build in test.
Confirmation Test Have bugs really been fixed? Double check!
Stress Testing How well does the application behave in harsh conditions?Experiment.
BU
ILD
FAST Test Each testable object has a simple test.Is the testable object stable enough to test
Regression Test Does application still work as before?Did we accidentally break something?
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 49
Smoke Testing
• Smoke test is run on a new build of software to make sure all functions operate well enough to continue testing
• “Turn on a new appliance at the store”
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 50
FAST Testing
• Functional Acceptance Simple Tests– Wide in breadth, low
in depth– Exercise every
function of the application at least once
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 51
Regression Testing
• Previously executed tests are re-executed against a new version of the application– Have code changes broken something that
used to work– Have we introduced new defects
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 52
Confirmation Testing
• Typically:– Tester confirms that the
fixed bug is really fixed in the appropriate software build
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 53
Stress Testing
• Testing operational characteristics of application within a harshly constrained environment– Limit processor– Limit memory– Limit disk space– Diminish access to
shared resources
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2009 Slide 54
Under Pressure
Chartered Session Based Testing
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 55
C. Northcote Parkinson
Parkinson’s Law:“…work expands so as to fill the
time available for its completion…”
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 56
Just In Time Testing
Exploratory Testing
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 57
Exploratory Testing
• James Bach (www.satisfice.com)
• General Functionality and Stability Test Procedure for Windows 2000 Application Certification
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 58
Mandate to explore
The object of your mission is to explore the Missouri river, & such principle streams of it, as, by its course and communicationwith the waters of the Pacific ocean...may offer the most direct & practicable water communication across this continent for thepurposes of commerce.
- Thomas Jefferson's letter to Meriwether Lewis, June 1803
William Clark Meriwether Lewis
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 59
Make intelligent decisions
Take notes about your decisions
Map out where you have been
Others can use the result
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 60
Chart as you explore
Further exploration yields a good idea of the state of the world!
One bit at a time
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 61
Exploration Notes
- Tabular- Chronological- Schematic- Point form- Concise
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 62
Exploratory Testing
• Test Cases – Not known in advance– Defined & executed “on the fly” while you learn
about the product
• Map Making Skills– Consistent note taking style– Practice
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 63
Exploratory Testing
• During test we must capture– Function, options or sub-functions being explored– Test cases attempted– Comments, notes, images or attachments – Hints, reminders and observations which may be useful to
future testers– Date, Platform, Build or Configuration under test– Name of person running test– Oracles, “strategy to assess correctness”– Other relevant details
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 64
An Exploratory Test Process
Kick Off
Prepare
Run
Complete
Review
Follow Up
Confirm Test ObjectiveEnsure context known
Ensure HW and SW OKAll tools available
Chunk of 90 to 120 minTest, Plan, Discover
Wrap upCollect all notes data
Review results withTest Lead
Reassess goalsPiece together map
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2009 Slide 65
Finished?
How do you know you are finished?
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2009 Slide 66
You know you are finished when
…
… the only bugs left are the ones are acceptable (based on your objective test team input) ...
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2009 Slide 67
… the only bugs left are the ones are acceptable (based on your objective test team input) ...
At least for now!
You know you are finished when
…
AmiBug.Com, Inc.
April 19, 2023 © Robert Sabourin, 2008 Slide 68
Bottom Line
• My experience is that it is better to omit a test on purpose than to skip it because you ran out of time or forgot about it!
Get
Sta
rted
Rig
ht • Systematically collecting, evaluating and triaging testing ideas helps me decide what not to test - at least for now?