ROI at the bug factory - Goldratt & throughput (2004)

www.qualtechconferences.com© Thompson information Systems Consulting Limited

1

ROI at the Bug Factory:Goldratt and Throughput

1st European Conference on Software Projects, Process & People01 - 03 December 2004 – Köln, GermanySoftware Projects Track session WP1

Neil ThompsonThompson information Systems Consulting Limited

www.TiSCL.com


2

1. Introduction• Primary objective of presentation – to explore some analogies between

processes for software development and implementation and those for manufacture (of physical things):– to see whether recent (and not-so-recent) revolutionary changes in

manufacturing have relevance for software– to relate such analogies to existing work in agile methods– to relate this thinking to the software development lifecycle whether agile

or not, and to verification, validation & testing (VV&T) – to help you in planning and managing projects & programmes

• Secondary objectives: – to use the concept of Return On Investment (ROI) as an introduction to

thinking about process effectiveness, efficiency and improvement– to use Goldratt’s Theory Of Constraints (TOC) as the basis of considering

manufacturing processes, and for its general thinking tools


3

What is ROI, why measure it?

• ROI: return / investment (usually expressed as a percentage):– “return” could be profit, (value – cost), (net benefit)…– “investment” may be expressed as net cost

• I’m a “crossover” from EuroSTAR (but wider background)• So I’ll start with the example of ROI for VV&T:

– to decide how many test environments to buy, and how life-like– to decide how many testers & QA people to hire– to estimate duration of testing– to justify budgets– to decide how to divide effort between testing & reviews/inspections– to focus training– to justify automation– to motivate staff?– to make managers look good?


4

Why ROI difficult for IT systems

• Although costs relatively easy to measure, not trivial• Benefits no longer about replacing manual systems; often are

enhancing market position, or opening new market, and those can be very difficult to forecast

• Misleading / unbalanced precision between costs & benefits• Misguided cost-cutting (eg through inadequate understanding of

benefits)• Excuses for not calculating ROI “properly”:

– “we’ve simply got to have it”– project / system too small to bother– it’s too difficult– wouldn’t be believed anyway

• Problem of “construct validity”: you may not really be measuring what you think you’re measuring Cem Kaner


5

ROI even more difficult for VV&T

• Partitioning difficulties:– how to distinguish as activities, when much is part of develop’t work– whether / how / why to distinguish by job title

• Biggest problem is quantifying benefits in terms of costs saved by preventing failures later:– Boehm’s exponential-type cost rise questioned for modern methods– isn’t as simple as “early=good”, because some bugs not (easily)

visible until later (though continuous integration helps)– how can we say fault A would have caused failure B at date X if it

had not been found by test level C?– even where live failure costs calculated, validity is questionable– how distinguish poor VV&T from “good VV&T where there are very

few bugs there to find”?


6

Also other complications, but let’s persist still

• Additional complications to using ROI for VV&T:– “Tyranny of numbers” two different books

– whether testing actually reduces risk or is merely (a misinterpretation of?) “Project Intelligence” Paul Gerrard

– is effectiveness / efficiency distinction feasible / useful?• But ROI thinking should still be helpful:

– still need to answer those questions, don’t we?• which VV&T activities to do, which not, which partially?• (coarse-tune) which people to use?• (fine-tune) how to maximise effectiveness & efficiency?

– so Agenda…


7

Agenda

• 2. Goldratt’s Theory Of Constraints• 3. The code factory and the bug factory• 4. Beyond the factory: the supply chain, and live systems• 5. The Systems Development LifeCycle as an

inventory network• 6. Conclusions & way forward


8

2. Goldratt’s Theory Of Constraints: what it is

• Now, “multifaceted management philosophy”, but originally emerged in early 1980s from new approach to materials scheduling in factories

• I already applied Goldratt’s thinking tools to process definition in software testing STAREast 2003

• Here looking wider, at whole SDLC, going back to Goldratt’s original arguments in manufacturing The Goal, The Race

• Evidence of TOC success in reducing inventory, lead times & cycle times; improving due-date performance & revenue

• Associated with Japanese improvements (eg Just In Time, Lean), but JIT came first, Goldratt was a response and a claimed improvement; Lean followed a few years later


9

Parts of TOC using here• Goal definition, eg make money:

– measures cash flow (survival), profit (absolute) & ROI (relative)• Throughput better for these measures than cost management• Adverse effect of inventory on competitive edge:

– it damages not only ROI & cash flow but also profit• Relationship of throughput to inventory & operating expense:

– throughput (T) = money in (should maximise)– inventory = money tied up (should minimise, but not damage T)– operating expense = money out (should minimise, but not damage T)

• Drum-Buffer-Rope as a technique to optimise inventory• Relationship to Synchronous Flow Manufacturing, Japanese

methods and other quality techniques


10

Applicability of TOC beyond manufacturing

• Military logistics• Marketing, sales & distribution• Project management• Measurements, human relationships, medicine etc• Using technology, eg assessing benefits of functionality• IT systems development:

– focussing of Goldratt’s Critical Chain on hi-tech projects Robert C Newbold

– methodology design Alistair Cockburn

– “Lean software development” Mary & Tom Poppendieck

– Agile management using Feature Driven Development David J Anderson


11

But software development isn’t like manufacturing?• Software isn’t like hardware• Intellect adds value less predictably than machines• The manufacturing part of software development is disk

duplication: “development” is really a design activity• People are more important than processes• Software development doesn’t repeat exactly, people are

always tinkering with the processes• Development involves discovery, production involves

reducing variation• But I say: does that make all the analogies worthless, or

do they just need interpreting? I suggest the latter…


12

3. The code factory and the bug factory

• No-waste factory

• Now here’s some waste: meeting/escalation (and loaded personal memories), or inventory?

Programming

Programming

a ab b’

c d

Documentedrequirements

Implicitrequirements

Meeting / escalation to agree

I I Acceptance tests

?

?

a b c Demonstrations &acceptance tests

Statedrequirements


13

Specs & unfinished software are inventory

• Specifications are generally not needed after go-live (I will come later to exceptions) so they are not end-product, they are work-in-progress (especially intermediate levels like functional & non-functional specs)

• Untested software, and even finished software not paid for, is also inventory

• Iterative lifecycles help if “adaptive” (product-based) rather than “transformational” (where specifications multiply!)

a b

ProgrammingProgramming

b’

a’

c

I

May includeredesign

Revised & new requirements


14

Full W-model bulges with inventory

Requirements Statement

Functional Spec.

Technical Design

Module Specs.

Code

Business objectives

V&V MS, + spec. UT

Unit test

Unit retest, fix & reg. test

Post-implementation review

Verify & Validate RS, + spec. AT

Acceptance test

Acc. retest, fix & reg. test

Sys. retest, fix & reg. test

Int. retest, fix & reg. test

System test

Integrat- ion test

Verify & Validate FS, + spec. AT

V&V TD, + spec. IT

Makeinto

Test against

Retest lowerlevelswhere

necessary

Static-check

Verify

Validate(incl. “QA”)


15

In a factory, small batches reduce inventory

Based on Goldratt, The race (North River Press 1986)

200Multi-batch(ie “iterative”)

(i) 1.3

(ii) 10.0

(iii) 1.0

(iv) 10.0

(v) 2.0

0 1 2 3 4 months

200 200 200 2001000

Single-batch(ie “waterfall”)

0 1 2 3 4 months

Inventory Inventory

Stages &units / hour


16

Why inventory damages competitiveness (i)

Goldratt’s objectives for competitiveness

My IT analogies

Product: 1. quality Quality & scope: 1. few defects now, reliable in future

2. engineering 2. amount & sophistication of functionality, non-functional attributes

Price: 3. higher margins Low-cost: 3. economical development & testing so flexible pricing

4. lower investment per unit

4. no environments panic, lower break- even point & flexibility to compete

Responsiveness: 5. due-date performance

Timely delivery: 5. rarely / never late

6. shorter quoted lead- time

6. rapid application development!


17

Why inventory damages competitiveness (ii)

Objectives for competitiveness

Advantages of low inventory (and/or iterative lifecycles!)

Product: 1. quality Defect detection & fixing sooner after their introduction, fewer recurrences

2. engineering Quicker realisation of improvements

Price: 3. higher margins Lower probability of needing overtime

4. lower investment per unit

Lower probability of needing additional machines (test environments)

Responsiveness: 5. due-date performance

More reliable materials forecasting (smoother staff recruitment)

6. shorter quoted lead- time

Shorter overall time (already shown on slide 15)


18

Throughput relationships, why good

Competitive edge Extended from Goldratt, The race (North River Press 1986)

High quality Good scopeLow people costsLow machine costsRarely lateRapid

InventoryOperatingexpenseThroughput

ROINet profit Cash flow


19

Drum-buffer-rope• Optimise throughput by:

– (1) drumbeat based on constraining stage (a) & orders (b)– (2) buffer to protect constraining stage from upstream disruptions– (3) rope to prevent leader extending gap on constraining stage

• For subassemblies feeding in, have additional buffers

assembly constrainingstage (a) ”leader”

buffer

orders (b)

subassembly buffer

raw materialsin

“troops marching” materials flow

(1)

(2)

(3)


20

Buffer management

x days

y hours

healthy

toocautious

toolean

interruptionof inventory

in

prematurerelease

of inventoryin


21

Tracing buffer holes back to process steps to fix

3 5

4

2 1


22

Relationship to other quality approaches

• To recap: Goldratt argues why DBR is better than JIT• In addition to TOC, use lean techniques to minimise waste• TOC project management (critical chain, see Newbold):

– deprecates multi-tasking– translates buffer management to contingency placement– recommends also time management & Pareto techniques– uses risk management to calculate buffer sizes

• Goldratt is more process-oriented than product- (as was Deming)


23

Is VV&T a factory?• VV&T produces (depending on your viewpoint):

– “project intelligence”;– risk reduction;– a mountain of test plans, scripts, logs, reports etc etc;– bug reports

• Customer does not want to pay for bugs• Bugs are imperfections on inventory rather than inventory

themselves; may not be many in the first place• “Removal of bugs” is not an easy end product to grasp,

and the developers do it anyway • So the code factory & the bug factory are two aspects of

the same thing; VV&T can help identify constraints


24

4. Beyond the factory: supply chain &

live systems• TOC/JIT says that functionality should be pulled by

customers rather than pushed by analysts / vendors (but why does this often not happen?)

• Software passing testing but not yet paid for is still WIP• Software development has its value chain• Poppendieck principles include “decide as late as

possible”: contradiction with Newbold?• Biggest point here is that most systems go on for years,

and need some documentation or staff continuity (this point receives insufficient attention often)


25

5. SDLC as an inventory network

• TOC suggests choosing SDLC with a view to what else is wanted other than working software, eg training material, maintenance documentation, regulatory audit

• This affects value of inventory• Cockburn uses TOC principles, of which one consequence is effect on

method• Another contradiction? Advice on handling slack• TOC process improvement can monitor buffers, and is not just for

agile methods• Can still think of VV&T ROI as tangibles & intangibles, even if we

can’t (yet?) calculate• But via the throughput route to ROI we can still seek to optimise


26

Inventory moving through SDLC

Amount offunctionality

Date

Systemtesting

Live andpaid-for

Acceptancetesting

Programming &unit testing

Integrationtesting

Requirements

Design

Specification

If lines not approx parallel, inventory is growing

Inventory inprocessoverall

Inventory inthis stageof process

Lead time forthis stageof process

Within each stage of testing, can subdivide by

pass/fail, bug states etc

Based on Anderson, Agile management for software engineering (Prentice Hall 2004)


27

SDLC as an inventory network (cont’d)

order(s)Acceptancetesting

Systemtesting

Integrationtesting

Unittesting

Programming

Requirements

raw materialsin Where is/are the

constraining stage(s)?

Where should buffersbe / not be?

Design

Specification

assembly

sub-assemblies


28

6. Conclusions and way forward

• Although original Goldratt ideas don’t translate directly to software development, after so much success and so many analogies, can’t ignore (and thinking tools useful for “anything”)

• Several of the later interpretations have plausible messages • Although a few authors have already applied TOC to agile

methods, I hope that by going back to the original principles in some detail I have widened consideration to potentially any SDLC, and provided a platform for me and others to develop this thinking further…


29

Way forward

Constraint-ledprocess definition& improvement

Context → “Methodologyper project”

project-informing

thinkingtools

Maintenance regime

Circumstance-driven

Publishedmethodologies(“Tailorable”?)


30

Summary

“How clean is your factory?”

Heavy industry, waste, burnout Light industry, lean, cool fumes


31

References & acknowledgements

• Main references:– Goldratt, Eli: The race & Necessary but not sufficient– Newbold, Robert: Project management in the fast lane– Mellis, Werner: Process & product orientation– Cockburn, Alistair: Agile software development– Poppendieck, Mary & Tom: Lean software development – DeMarco & Lister: Waltzing with bears– Anderson, David: Agile management for s’ware engineering– Boehm & Turner: Balancing agility & discipline

• Acknowledgements:– Jens Pas EuroSTAR 1998, metrics– Greg Daich STAREast 2002, documentation superstitions


32

Contact details

Neil Thompson

[email protected]

www.TiSCL.comQuestions?

23 Oast House CrescentFarnham, Surrey, EnglandGU9 0NP, United Kingdom

phone +44 (0)7000 NeilTh (634584) or +44 (0)7710 305907

ROI at the bug factory - Goldratt & throughput (2004)

Technology

Transcript of ROI at the bug factory - Goldratt & throughput (2004)