©2003 Systeme Evolutif LtdSlide 1 we’ll change the way you think about testing What Is the Value...
-
Upload
roman-bannister -
Category
Documents
-
view
213 -
download
0
Transcript of ©2003 Systeme Evolutif LtdSlide 1 we’ll change the way you think about testing What Is the Value...
©2003 Systeme Evolutif Ltd
Slide 1we’ll change the way you think about testing
What Is the Value of Testingand How Can We Increase
It?
Paul GerrardTechnical DirectorSysteme Evolutif
Systeme Evolutif Limited9 Cavendish PlaceLondon W1G 0QD, UKTel: +44 (0)20 7636 6060email: [email protected]://www.evolutif.co.uk/
©2003 Systeme Evolutif Ltd
Slide 3we’ll change the way you think about testing
Themes
“[A cynic is] A man who knows the price of everything but the value of nothing”
Oscar Wilde (1854-1900), Lady Windermere’s Fan.
“To win without risk is to triumph without glory”
Pierre Corneille (1606-84), The Cid.
©2003 Systeme Evolutif Ltd
Slide 4we’ll change the way you think about testing
Agenda Let’s play Marketing Quantity, Quality, Process or Product –
Where is the Value? The Most Valuable Product of Testing is
Project Intelligence Influences on the Value of Testing Increasing the Value of Testing Process “Improvements” and the Value of
Testing Summary.
©2003 Systeme Evolutif Ltd
Slide 5we’ll change the way you think about testing
Let’s Play Marketing We need to consider:
– Who are the customers of testing?– What exactly does the customer value?– How can we increase the value of our testing?– How can we increase their appreciation of our
testing?– How do we select an “improvement” to make?– How do we judge the value of an
“improvement”?
©2003 Systeme Evolutif Ltd
Slide 6we’ll change the way you think about testing
Quantity, Quality, Process or Product – Where is the
Value?
©2003 Systeme Evolutif Ltd
Slide 7we’ll change the way you think about testing
Mine clearing According to captured records, there
are 100 mines in that minefield We have identified and removed
seven mines I crossed it once but don’t remember
how
Is it safe?
©2003 Systeme Evolutif Ltd
Slide 8we’ll change the way you think about testing
Test activity and information
I ran some tests– But I can’t remember what happened or how I could
repeat those tests– Was this testing valuable?
I ran a hundred tests and the system failed 23 times– I can’t remember how I did it or where the system
failed– Was this testing valuable?
The ACTIVITY of testing only has VALUE if it generates INFORMATION (this is the deliverable).
©2003 Systeme Evolutif Ltd
Slide 9we’ll change the way you think about testing
How much detail? Which is best?
– “We ran lots of tests”– “We found lots of faults”– “10% of our tests caused the system to fail”– “This test case caused a failure”– “This set of test steps caused the system to fail at
precisely this point, in precisely this way…” Test records should be detailed enough to
plan, design, implement, execute tests and interpret test results and reproduce failure– AND NO MORE.
©2003 Systeme Evolutif Ltd
Slide 10we’ll change the way you think about testing
Quantity or quality? Which would you pay more for?
– 100 page test plan or 200 page test plan?– 100 incident reports or 200 incident reports?– 100 requirements faults or 100 coding faults?– 100 random tests or 100 ‘designed’ tests?
Quantity is one thing, but the nature and quality of the information and the way it was prepared makes a huge difference to its value.
©2003 Systeme Evolutif Ltd
Slide 11we’ll change the way you think about testing
Testing generates “project intelligence”
If you don’t know where you are, a map won’t help If you don’t know the status of the deliverables,
the best plan won’t help you manage your project A project is like driving off-road in the dark
– Testing provides the headlights for the journey Testing provides data on the status of deliverables
and generates “PROJECT INTELLIGENCE” Stakeholders and management need intelligence
to make decisions, command and control the project.
©2003 Systeme Evolutif Ltd
Slide 12we’ll change the way you think about testing
Does testing ‘change’ anything?
Product TestingProduct
(unchanged)
ValuableStuff
Testing doesn’t change the product in any way. We do it to get the ‘valuable stuff’.
©2003 Systeme Evolutif Ltd
Slide 13we’ll change the way you think about testing
What is the valuable stuff? Test deliverables (infrastructure,
test plans, specifications, scripts, data, results etc…)
TEST RECORDS – test documentation (but do only testers read it?)
PROJECT INTELLIGENCE – rarely documented as such though.
©2003 Systeme Evolutif Ltd
Slide 14we’ll change the way you think about testing
Military intelligence?
©2003 Systeme Evolutif Ltd
Slide 15we’ll change the way you think about testing
The Most Valuable Product of Testing is Project
Intelligence
©2003 Systeme Evolutif Ltd
Slide 16we’ll change the way you think about testing
Documentation PRODUCTS
– Test Strategy– Test plans, designs, specifications– Procedures/scripts, expected results– Actual results– Incident reports– Log– End of Phase Report– Etc.
TANGIBLE, BUT LESS VALUABLE.
©2003 Systeme Evolutif Ltd
Slide 17we’ll change the way you think about testing
Test infrastructure PRODUCTS
– Test process– Configured test hardware– Test databases– Tools– Manual and automated tests– Skilled resources– Etc…
TANGIBLE, USUALLY AN OVERHEAD– (unless these are required deliverables).
©2003 Systeme Evolutif Ltd
Slide 18we’ll change the way you think about testing
TEST RECORDS DATA
– How the product SHOULD work (test specs)– How the product ACTUALLY works (test results)– Where the product is STABLE (incident trends)– Where the product is UNSTABLE (incident trends)– What they have COVERED in the product (test log)– What they have NOT COVERED (test plan – test log)– How (and where) the product FAILS (incidents)
TANGIBLE AND VALUABLE, BUT…– (valuable to testers, developers, NOT management).
©2003 Systeme Evolutif Ltd
Slide 19we’ll change the way you think about testing
PROJECT INTELLIGENCE KNOWLEDGE OF PROJECT RISKS/BENEFITS
– RISKS» Requirements are poor and/or unstable» Less of the product is ready, product is not as good as
planned» They should have been less ambitious, done it differently» The project will slip
– BENEFITS» Some features/benefits are available» Other features/benefits are blocked by risk uncertainty
INTANGIBLE AND VALUABLE– Crucial to management and stakeholders– Intelligence often arrives TOO LATE TO BE OF USE.
©2003 Systeme Evolutif Ltd
Slide 20we’ll change the way you think about testing
Where the value of testing lies
LOW VALUE– Documentation– Test infrastructure** UNLESS
these are specific objectives or deliverables of project
HIGH VALUE– Test Records– Project Intelligence
We can increase the value of testing MOST by:• Improved INTELLIGENCE• Providing INTELLIGENCE EARLIER.
©2003 Systeme Evolutif Ltd
Slide 21we’ll change the way you think about testing
Influences on the Value of Testing
©2003 Systeme Evolutif Ltd
Slide 22we’ll change the way you think about testing
Intangible value, tangible cost
Testers think testing is good and has undeniable value, but we are not independent!
Not everyone thinks that (software suppliers, in-house developers?)
If you are not a tester, the value seems intangible– But the cost IS tangible– Exercise, flossing, house maintenance, life insurance,
testing can all be put off till another day– If you don’t do it now you store up problems for later
A bit like a lifestyle choice – it’s human nature to put things off.
©2003 Systeme Evolutif Ltd
Slide 23we’ll change the way you think about testing
Key factors in the value of testing
Adding value depends on how well we achieve and communicate:1. Coverage2. Incidents3. Risk-based testing4. Benefits-based testing
Time to deliver that value depends on the test process (more later)
Willingness of the organisation to accept and act upon project intelligence is key (more later).
©2003 Systeme Evolutif Ltd
Slide 24we’ll change the way you think about testing
Coverage Breadth of coverage
– Are all aspects of the product tested? Depth of coverage
– Are critical aspects of the product tested more thoroughly?
Can the measure be applied systematically? Is the measure relevant to the faults to be
detected or the risks of concern? Can we communicate coverage in a
meaningful, understandable way?
©2003 Systeme Evolutif Ltd
Slide 25we’ll change the way you think about testing
Incidents Are the failures of most concern
being found and logged as incidents? Can these failures be reproduced
easily? Are the incidents being addressed in
the right order? Are the incidents of most concern
being resolved quickly?
©2003 Systeme Evolutif Ltd
Slide 26we’ll change the way you think about testing
Risk-based testing Are product risks identified early in the project? Are stakeholders involved in the decision to rank
these risks? Is the decision to include risks in scope for
testing done by consensus? Is the decision to de-scope risks done by
consensus? Is test design driven by the need to address risk? Is test progress reporting based on risks
addressed?
©2003 Systeme Evolutif Ltd
Slide 27we’ll change the way you think about testing
Benefit-based testing Does the project have cardinal
objectives (benefits) defined? Are all risks related to the benefits? Are tests related to the risks that
block the benefits? Is test progress reporting based on
the benefits that are threatened or available?
©2003 Systeme Evolutif Ltd
Slide 28we’ll change the way you think about testing
Increasing the Value of Testing
©2003 Systeme Evolutif Ltd
Slide 29we’ll change the way you think about testing
Contribution of the test process
Does testing generating valuable intelligence? Does testing generate intelligence early
enough? Is testing flexible enough to react to new risks? Are the test communication channels clear? Improvement types:
– Effectiveness– Efficiency– Perceived value of testing.
©2003 Systeme Evolutif Ltd
Slide 30we’ll change the way you think about testing
Effectiveness improvements Improvements to “intellectual” test
activities enhance the quality of intelligence Implementing these activities earlier make
intelligence available earlier Examples:
– Formal test design improves coverage, and improves the quality of test records
– Improvements that promote earlier test activities tend to make the value available earlier (e.g. more reviews).
©2003 Systeme Evolutif Ltd
Slide 31we’ll change the way you think about testing
Efficiency improvements These reduce the cost of doing testing Improvements to the mechanical or clerical
aspects of our test process make us more efficient (but don't affect the value of testing)
For example– Using a tool to automate tests might make it
cheaper to do a task, but does it add value?– It might save time, allowing us to do other useful
activities Ultimately, effectiveness improvements are
more valuable.
©2003 Systeme Evolutif Ltd
Slide 32we’ll change the way you think about testing
Organisational culture Willingness to accept and act on the
intelligence that testers generate is key
Primarily a cultural issue If the intelligence isn’t valued by the
culture - it doesn't matter what value it has
We need to EDUCATE our customers.
©2003 Systeme Evolutif Ltd
Slide 33we’ll change the way you think about testing
Increasing the perceived value of testing
No such thing as “absolute value” of testing We must work on the PERCEIVED VALUE to the
customers of our testing:1. Improve the value in line with their
expectations– Provide richer, deeper, focused, relevant information– “Create more complex, subtle, interesting wine”
2. Increase their appreciation of what we do– Educate them to understand testing products– “Train their palate to detect the complex flavours”
Improvements must address both issues.
©2003 Systeme Evolutif Ltd
Slide 34we’ll change the way you think about testing
Process “Improvements” and the Value of Testing
©2003 Systeme Evolutif Ltd
Slide 35we’ll change the way you think about testing
Test process or test product assessment?
Assessments of the “Test Process” may miss the point
“Process healthchecks” are useless because they don’t focus on the VALUE of test deliverables
Assessment of the perceived value of Test Products must be a key driver
We must focus on process changes that:– add value to the test products– enhance the perceived value of test products– make test products available earlier.
©2003 Systeme Evolutif Ltd
Slide 36we’ll change the way you think about testing
The challenges of testing improvement
Barriers are organisational, personal, not technical:– changing management perceptions so they support
testing and improvement– overcoming management and practitioner
resistance to change– design and implementation of workable processes
and management controls People, not tools, implement test strategies Change is easier if people see VALUE being
added.
©2003 Systeme Evolutif Ltd
Slide 37we’ll change the way you think about testing
Problems or symptoms? Need to separate problems from symptoms:
– “Management doesn't understand the value of testing”
– “The cost of testing is high but difficult to pin down”– “Developers, testers, users never trained”– “The quality of the product delivered into testing is
poor, so takes longer to system test”– “The perceived value of testing is low”
Need to understand symptoms to identify the source of problem and fix the problem
Improvements often tackle the symptoms only.
©2003 Systeme Evolutif Ltd
Slide 38we’ll change the way you think about testing
Right improvement mix is key
A mix of improvements is most likely to be required:– management awareness– tester training– developer training– improved definition of the test stages and
their objectives– measurement of the quality of the product at
each stage– etc. etc.
©2003 Systeme Evolutif Ltd
Slide 39we’ll change the way you think about testing
Constraints and expectations
Not all improvements a good idea straight away– some improvements are expensive– some save time, but bring dramatic change– some improve the quality of the testing, but take
longer to implement– some tidy up ‘loose ends’ but have negligible value
Very few improvements save time, improve quality, cause minimal change and pay back after two weeks
Need to set/agree clear management expectations.
©2003 Systeme Evolutif Ltd
Slide 40we’ll change the way you think about testing
Do tools increase the value of testing?
With tool vendors present, I’d better be careful!
Need to distinguish between– Quantity, quality and the information provided
A tool might allow you do to testing faster, but it probably won’t increase the value of tests
It might make a task interesting, so a valuable task gets done
It might give you more time to do other valuable tasks.
©2003 Systeme Evolutif Ltd
Slide 42we’ll change the way you think about testing
How to increase the value of testing
The value of testing lies in the project intelligence it generates
Produce documentation with purpose:– To allow testers to do their job (THE MEANS)– To communicate intelligence (THE END)
Process ‘improvements’ selected to deliver better intelligence, earlier, faster and cheaper
You need to train your customers to appreciate the project intelligence that testers produce.
©2003 Systeme Evolutif Ltd
Slide 43we’ll change the way you think about testing
Themes revisited“A poor tester knows the challenge and cost of testing but not its value”
– Don’t be cynical – testing isn’t just a costly activity to be gotten through, testers should aim to deliver value
“To win without risk is to triumph without glory”– Risks are inevitable - the testers’ role isn’t to avoid
them, but inform management and reduce their uncertainty
– Satisfaction (and recognition) comes from overcoming risks, not doing the mundane tasks of development
– Make your management aware of software product risks and report progress towards their goal, not just incidents.
Testers deal in success, not just failure.