The ROI of TestingThe ROI of Testing

Presented By:Shaun BradshawQuestcon Technologies

sbradshaw@questcon.com

The ROI of TestingThe ROI of Testing

Presented By:Shaun BradshawQuestcon Technologies

sbradshaw@questcon.com

Quality - Innovation - VisionQuality - Innovation - Vision

Slide Slide 22

Return on InvestmentReturn on Investment

Slide Slide 33

Return on Investment (ROI)Return on Investment (ROI)

• What is ROI?ROI is a calculation that attempts to determine

the actual or perceived future value of an

expense or investment

By calculating ROI, an organization can assess

whether the expense/investment is justified by

the resulting savings/revenue

Slide Slide 44

Return on Investment (ROI)Return on Investment (ROI)

• How is ROI calculated?In its most basic form ROI is calculated as

follows:

ROI = (Benefits – Costs)/Costs

That is, the financial benefit after an investment

or improvement is made minus the cost of the

investment or improvement, calculated as a

percentage of those costs Benefits• Revenue generated• Cost reduction• Cost avoidance• Productivity

improvements

Costs• Labor expense• Time expense• Tool expense

Slide Slide 55

• What makes testing valuable? How do we get to a positive ROI?

Produces information reliably grounded in

observed system behavior [1]

– Functions like a credit check [1] – Exposes risk and improves decision-making

Extends the life of the software

Improves the development process

Increases end-user confidence and satisfaction

Return on Investment (ROI)Return on Investment (ROI)

Slide Slide 66

Defect Preventionvs.

Defect Detection

Defect Preventionvs.

Defect Detection

Slide Slide 77

• Importance of TestingOrganizations rely on testers to do QA

We cannot “test” quality into an application

Test needs to be the “backdoor to QA”

Requires a shift in the testing paradigm

Test Finding

Defects

Defect Prevention vs. Defect Detection Defect Prevention vs. Defect Detection

Measuring Quality

Slide Slide 88

• Rules & Results of Defect Prevention

60% Rule [2]

Percentage of defects

introduced in require-

ments and design

85% Rule [2]

Percentage of defects

removed through

reviews, inspections

and testing

368:1 Rule [3]

Defect Prevention vs. Defect Detection Defect Prevention vs. Defect Detection

Slide Slide 99

Design Code Test ProductionRequirements

30

30

$0

$0

0

30

60

0

$0

$0

30

90

0

$0

$0

10

100

85

$166,600

$166,600

0

15

$378,700

15

$212,100

# of defects introducedAccumulated defects

Defects removed

Cost

Accumulated Cost

X 5X 10X 50X 368X+

Testing Only

Assume X is $100 &

100 total defects

Defect Prevention vs. Defect Detection Defect Prevention vs. Defect Detection

Test Only Costs

0

50000

100000

150000

200000

250000

300000

Require

ments

Test Only Costs

0

50000

100000

150000

200000

250000

300000

Require

ments

Design

Test Only Costs

0

50000

100000

150000

200000

250000

300000

Require

ments

Design

Code

Test Only Costs

0

50000

100000

150000

200000

250000

300000

Require

ments

Design

CodeTes

t

Test Only Costs

0

50000

100000

150000

200000

250000

300000

Require

ments

Design

CodeTes

t

Produc

tion

Slide Slide 1010

Design Code Test ProductionRequirements

30

30

$2,600

$2,600

26

30

34

29

$14,500

$17,100

30

35

30

$30,000

$47,100

10

15

13

$65,000$112,100

0

2

$185,700

2

$73,600

Full QA

Assume X is $100 &

100 total defects

Defect Prevention vs. Defect Detection Defect Prevention vs. Defect Detection

X 5X 10X 50X 368X+

# of defects introducedAccumulated defects

Defects removed

Cost

Accumulated Cost

0

50000

100000

150000

200000

250000

300000

Require

ments

Design

CodeTes

t

Produc

tion

Test Only Costs Full QA

0

50000

100000

150000

200000

250000

300000

Require

ments

Design

CodeTes

t

Produc

tion

Test Only Costs Full QA

0

50000

100000

150000

200000

250000

300000

Require

ments

Design

CodeTes

t

Produc

tion

Test Only Costs Full QA

0

50000

100000

150000

200000

250000

300000

Require

ments

Design

CodeTes

t

Produc

tion

Test Only Costs Full QA

0

50000

100000

150000

200000

250000

300000

Require

ments

Design

CodeTes

t

Produc

tion

Test Only Costs Full QA

Slide Slide 1111

Quantitative BenefitsQuantitative Benefits

Slide Slide 1212

• Reduced Defect Repair CostsTest involvement earlier in the lifecycle reduces repair costsAutomation improves defect detectionGood test practices increase detection likelihood

Quantitative Benefits Quantitative Benefits

Full QATest OnlyAd Hoc Testing

$0

$100,000

$200,000

$300,000

$400,000

$500,000

$600,000

Slide Slide 1313

• Decreased Production CostsDefects in production can be significantly higher than 368x -

the “Annuity Nightmare”– Small rounding error goes undetected at a major financial institution– Months until discovered– Code repair is cheap– Production cost – NOT CHEAP!

• Increased Revenue & ProfitsReduced time-to-market for functioning software

Increased market share

Higher customer retention/goodwill

Reduced maintenance costs (easier to enhance)

Quantitative Benefits Quantitative Benefits

Slide Slide 1414

Calculating Test ValueCalculating Test Value

Slide Slide 1515

• Defect Injection RateTotal number of defects introduced into an application

When each defect was introduced (using Root Cause

Analysis)

Number of defects introduced in each phase of the SDLC

Calculate the defect injection rate as:

# of defects introduced

total defects

Key Metrics to Calculate Test Value Key Metrics to Calculate Test Value

Slide Slide 1616

• Defect Repair CostHourly rate of each resource involved in the repair (PM, BA,

Developer, Tester, etc.)

Defect repair time for each resource by phase of the SDLC

Number of defects repaired in each phase of the SDLC

Calculate the defect repair costs as shown in the example*:

*Data in the example was drawn from [4].

Key Metrics to Calculate Test Value Key Metrics to Calculate Test Value

Slide Slide 1717

• “Test” EffectivenessNumber of defects found during “testing” activities

during any phase of the SDLC

Total number of defects found including in production

(use a “warranty period”)

Calculate test effectiveness as:

# of defects foundphase

total # of defects found

Key Metrics to Calculate Test Value Key Metrics to Calculate Test Value

Slide Slide 1818

ROI ExampleROI Example

Slide Slide 1919

ROI Example ROI Example

Slide Slide 2020

Established a test team made up of:• 1 test lead• 1 test analyst• 1 test consultant

Total 1st year investment: $237,500

Established a test team made up of:• 1 test lead• 1 test analyst• 1 test consultant

Total 1st year investment: $237,500

ROI Case Study ROI Case Study

Slide Slide 2121

Conducted 2 projects in first 8 monthsRealized $173,000 savings in project-over-project defect repair costsCalculated savings do not include other benefits such as:

•Reduced production downtime•Reduced maintenance costs• Increased customer satisfaction & trust

Anticipated 46% ROI at the end of year 1

Conducted 2 projects in first 8 monthsRealized $173,000 savings in project-over-project defect repair costsCalculated savings do not include other benefits such as:

•Reduced production downtime•Reduced maintenance costs• Increased customer satisfaction & trust

Anticipated 46% ROI at the end of year 1

ROI Case Study ROI Case Study

Slide Slide 2222

Questions Questions

sbradshaw@questcon.com336-273-2428

Slide Slide 2323

References References

1. Bullock, James. “Calculating the Value of Testing.” Software Testing & Quality Engineering: May/June, 2000. <www.stickyminds.com/>

2. Jones, Capers. "Software Cost Estimating Methods for Large Projects." CrossTalk: April, 2005. <www.stsc.hill.af.mil/crosstalk/2005/04/0504Jones.html>

3. Dabney, JB. "Return on Investment of Independent Verification and Validation Study Preliminary Phase 2B Report." Fairmont, WV: NASA IV&V Facility, 2003. <sarpresults.ivv.nasa.gov/ViewResearch/289/24.jsp>

4. US Dept. of Commerce, National Institute of Standards & Technology (NIST). “Planning Report 02-3: The Economic Impacts of Inadequate Infrastructure for Software Testing.” Technology Program Office, Strategic Planning & Economic Analysis Group. May, 2002.<www.nist.gov/director/prog-ofc/report02-3.pdf>