The Value-Adding Tester

ConfidentialPA12013-12-131

The Value-Adding TesterHow a Tester Adds Value to an Organization or Project


Introduction

▪ The Value-Adding Tester adds value to the project or organization

▪ The Value-Adding Tester understand and reduces costs

▪ The Value-Adding Tester does not detract value

▪ To be a value-adding tester it is required to continuously evaluate value and cost

▪ How does a tester add and detract value, and what are the costs?


How to Add Value - Overview

Fixed DefectsInformation to Stakeholders

Development Support

Mandatory Tests


Fixed Defects

▪ One of the major ways a tester can add value is by finding defects in the product which are actually fixed at some point during the product life cycle

▪ The value of a defect found can be quantified by estimating how the defect would impact sales, and if the customer returns the product because of the defect


Information to Stakeholders

▪ Another way to generate value for a tester is to provide stakeholders with information and material for tollgates, milestones and decision points

▪ The value is harder to quantify but can be divided into two parts:

▪ Helping stakeholders taking informed decisions

▪ Allowing stakeholders to feel more confident in their decisions


Mandatory Tests

▪ In many cases different governing bodies require that certain criteria are met for a product to be allowed to be sold

▪ Meeting these criteria often require mandatory tests that must be executed according to strict requirements

▪ Specific customers can also have mandatory requirements that must be tested to be able to sell to those customers

▪ The value a tester provides by executing these tests can be quantified by comparing how much it would cost to outsource this to an accredited outsourcing partner


Development Support

▪ A tester can also provide value by increasing the efficiency of the development team

▪ This can be done by providing a robust, easy-to-use test framework, or by creating and maintaining automated integration and regression test suites which the developers can use

▪ The tester can also offer support by helping with unit test plans and strategies

▪ Yet another example could be pushing for and educating in testability, which can also drive efficiency


How to Detract Value - Overview

Irrelevant Defects

Irrelevant Information

Inefficient Test Tools

Low Quality Reports


Low Quality Reports

▪ Testers should not create defect reports that

▪ Are hard to understand because of bad language

▪ Are duplicates of already existing defect reports

▪ Lack information that is critical to understand the defect

▪ Lack critical attachments such as log files or screen shots

▪ Etc.

▪ This will lead to increased report handling and analysis effort for developers

▪ If testers create reports which cannot be understood due to improper structure, language, formatting, etc., this also costs additional analysis effort for stakeholders


Irrelevant Defects

▪ When a tester reports a defect this has to be analyzed, prioritized and handled in different ways

▪ If the defect is not fixed, and the information stored in the defect is not used by stakeholders, submitting the defect actually detracts value instead of adding

▪ Many irrelevant defects could however together provide valuable data by revealing trends or problem areas

▪ The value detraction can be reduced by handling less relevant defects differently than higher priority issues


Irrelevant Information

▪ When testers provide reports to stakeholders, the information presented must be beneficial and help the stakeholders to take required decisions

▪ Presenting data that is not relevant or even misleading is not only not valuable, but can be very costly

▪ Burying key information in a mountain of data

▪ A report that states “99 % Pass Rate” as the main conclusion, when the remaining 1% represents critical quality issues in the product, will not help stakeholders make the correct decisions


Inefficient Test Tools

▪ The cost that testers impose on stakeholders with inefficient tools

▪ When testers use tools that impact others, such as developers, project managers, or other stakeholders, in a negative way, this detracts value

▪ One example could be inefficient test frameworks or automation tools which cause developers to write inefficient tests

▪ Another example could be a reporting tool which is difficult for stakeholders to extract reports from, or the tool generates reports which take time to analyze and understand


Costs - Overview

Test Design & Execution

Test Planning Test Reporting

Test Tools & Frameworks

Administrative Overhead


Test Design & Execution

▪ Designing, executing, maintaining and porting tests

▪ Execution Effort

Exploratory Test Scripted Test Automated Test

Automated Test Scripted Test Exploratory Test

▪ Design Cost


Test Planning

▪ Planning test activities and creating corresponding test artifacts such as test plans

▪ Setting scope for different test activities

▪ Risk analysis as impact to scope selection

▪ Probability of failure▪ Technical risk analysis

▪ Impact of failure▪ Business risk analysis


Test Reporting

▪ Creating test reports

▪ Test result metrics

▪ Qualitative summaries/product stories

▪ Creating defect reports

▪ Analysis of automated test results


Test Tools & Frameworks

▪ Creating new tools and test frameworks

▪ Cost of buying test tools and frameworks

▪ Integrating new tools

▪ Maintaining tools and frameworks


Administrative Overhead

▪ How much time does the tester actually spent on design, analysis, execution, tools and reporting, and how much time does the tester loose to administrative overhead?

▪ Getting the right software artifacts for test, and understand what to actually test

▪ Coordinating between testers, developers, and organizations

▪ Non-value adding meetings, etc.


Net Value of a Tester

Cost Value DetractionValueNet Value


How do we increase value and decrease cost and value detraction?

▪ We want to maximize value gain, and minimize costs and value detraction

▪ How we do this depends heavily on context

▪ However we can still give some general guidelines

▪ Of course there are many other ways to reduce costs and add value, but these are some suggestions



▪ Evaluate test artifacts and remove the non-value adding ones

▪ 10-minute Test Plan – don’t create extensive test plans that no one actually uses or updates

▪ Very detailed scripted test cases cost more than they add value most of the time

▪ Test Strategy documents that no one reads or uses should not be created at all▪ However the discussions which lead to the creation of the documents

are still valuable and important to have



▪ Evaluate tools among all different stakeholders to secure that they are efficient

▪ Even if a tool is easy to use for some purposes, it could impose large costs for other stakeholders to use

▪ A test administration tool that is easy to record data in, may be very costly to generate reports from

▪ Evaluate the actual value gain of the automated test framework

▪ Don’t just calculate the execution effort saved▪ Many different costs: Design, Analysis, Maintenance, Porting

▪ Actual gains: Defects Found, Mandatory Tests, Information and Support to Developers



▪ Secure that the information and defects testers report is what stakeholders actually need

▪ Defects must be fixed

▪ Information must be used

▪ Make sure that testers actually work with testing, and not spend their time on everything else

▪ Let testers list what they spend their time on, and try to reduce administrative overhead and let the testers actually work with test



▪ Proper risk analysis to support scope selection

▪ Risk-based testing

▪ Should be used when the reduction in cost for test execution is larger than the increase in cost for test planning that the risk analysis adds

▪ Efficiency gain of risk-based testing is very dependant on the complexity of the system under test

▪ Show the cost that the system complexity adds to testing – this can drive actions to reduce the complexity of the system


Conclusion

▪ The Value-Adding Tester continuously evaluates what value is added, what value is detracted, and the costs for providing this value

▪ There are many ways to add value, but there are also many ways to detract value and either way there is an associated cost

▪ Understanding the values and costs is critical to become efficient

▪ Understanding must be supported by actual metrics, and not only gut feelings

The Value-Adding Tester

Technology

Transcript of The Value-Adding Tester