Post on 09-Feb-2022
© Quantitative Software Management, Inc. All rights reserved. Slide #1
Quantitative Software Management
Using Metrics to Develop aSoftware Project Strategy
Donald M. BeckettQSM, Inc.
2000 Corporate Ridge, Suite 900McLean, VA 22102
(360) 697-2640, fax: (703) 749-3795e-mail: don_beckett@qsm.com
http://www.qsm.com
© Quantitative Software Management, Inc. #2
OutlineOutline
•• OverviewOverview
•• Measurement, Expense or InvestmentMeasurement, Expense or Investment
•• State of the Industry: Project EstimationState of the Industry: Project Estimation
•• Staffing and ScheduleStaffing and Schedule
•• Understanding TradeUnderstanding Trade--offsoffs
•• ConclusionConclusion
•• Questions?Questions?
© Quantitative Software Management, Inc. #3
OverviewOverview
Does this sound familiar?
© Quantitative Software Management, Inc. #4
Measurement: Expense or InvestmentMeasurement: Expense or Investment
•• Software measurement (and process Software measurement (and process improvement) are viewed as improvement) are viewed as expensesexpenses: : OverheadOverhead–– Lean, agile organizations want to reduce overheadLean, agile organizations want to reduce overhead
–– But, how do organizations become “lean & agile”?But, how do organizations become “lean & agile”?
•• Part of cost of doing business Part of cost of doing business –– 3 3 –– 5% on average5% on average
–– Project management averages 14%Project management averages 14%
© Quantitative Software Management, Inc. #5
Measurement: Expense or InvestmentMeasurement: Expense or Investment
•• What does software measurement provide?What does software measurement provide?1.1. Knowledge of an organization’s capabilitiesKnowledge of an organization’s capabilities2.2. Identifies patterns and trends (Strengths to leverage and Identifies patterns and trends (Strengths to leverage and
weaknesses to correct)weaknesses to correct)3.3. Insight into projects in time to make effective midInsight into projects in time to make effective mid--stream stream
correctionscorrections4.4. Ability to benchmark against competition or “the industry” in Ability to benchmark against competition or “the industry” in
quality, productivity, and time to marketquality, productivity, and time to market5.5. Quantitative basis for evaluating project and organizational Quantitative basis for evaluating project and organizational
performanceperformance
•• Improves ability to meet commitments, avoid Improves ability to meet commitments, avoid pitfalls, and evaluate tradepitfalls, and evaluate trade--offsoffs
© Quantitative Software Management, Inc. #6
State of the Industry: Project State of the Industry: Project EstimationEstimation
•• Software estimates are Software estimates are notnot project plansproject plans
•• Estimates contain uncertainty about two key Estimates contain uncertainty about two key components:components:–– Scope of the requirements (project size)Scope of the requirements (project size)
–– Team productivityTeam productivity
© Quantitative Software Management, Inc. #7
The Cone of UncertaintyThe Cone of Uncertainty
• Not enough information is available early in the development lifecycle to make accurate estimates
• Precision is not accuracy
Low level requirements
Project Definition
© Quantitative Software Management, Inc. #8
Actual vs. Estimated EffortActual vs. Estimated Effort
Effort Growth
0
10
20
30
40
50
60
Smaller At Size Larger
Actual vs. Estimated Effort
Perc
ent
© Quantitative Software Management, Inc. #9
Actual vs. Estimated ScheduleActual vs. Estimated Schedule
Schedule Growth
0
10
20
30
40
50
60
Smaller At Size Larger
Actual vs. Estimated Schedule
Perc
ent
Percent
© Quantitative Software Management, Inc. #10
Actual vs. Estimated SizeActual vs. Estimated Size
Size Growth
0
10
20
30
40
50
60
70
80
90
100
Smaller At Size Larger
Actual vs. Estimated Size
Perc
ent
© Quantitative Software Management, Inc. #11
In SummaryIn Summary
•• Average schedule growth is 8%Average schedule growth is 8%
•• Average cost/effort growth is 16%Average cost/effort growth is 16%
•• Average size growth is 15%Average size growth is 15%
•• So how can we use this information to create more So how can we use this information to create more accurate estimates?accurate estimates?
© Quantitative Software Management, Inc. #12
Modeling Increased SizeModeling Increased Size
•• Create best project estimate based on proposed Create best project estimate based on proposed sizesize–– Use historically based productivityUse historically based productivity
–– Account for project constraints (staff, effort, schedule)Account for project constraints (staff, effort, schedule)
•• Create estimate based on 15% size growthCreate estimate based on 15% size growth–– Does this account for projected schedule & effort growth?Does this account for projected schedule & effort growth?
© Quantitative Software Management, Inc. #13
S taffin g & P rob ab ility Ana lys is
Avg S ta ff L i fe C ycle (p e o p le )< 500 F P projec t>
1 2 3 4 5 6 7 8 9 10 11O c t'07
N ov Dec Jan'08
Feb M ar A pr M ay Jun Jul A ug S ep
0
1
2
3
4
5
6
7
Avg S
taff Life Cycle (people)
87654321
M iles tones 0 - C SR 1 - S R R 2 - H LD R 3 - LLDR 4 - C U T 5 - IC 6 - S T C 7 - U AT 8 - F C R 9 - 99R 10 - 99 .9R
M iles tones 0 - C SR 1 - S R R 2 - H LD R 3 - LLDR 4 - C U T 5 - IC 6 - S T C 7 - U AT 8 - F C R 9 - 99R 10 - 99 .9R
S O L U TI O N PAN EL - <500 F P p ro ject >
D ura tionEffo r tC ost
Pea k S ta ffM T T D
Star t D a te
C & T6 .729
493 .45 .7
1 .82312 /23 /2007
L if e C yc le9 .437
643 .65 .7
1 .82 310 /1 /2007
M o nth sP M$ ( K)pe op leD ay s
PI =16 .5 M B I =3 .6 E ff F P =500
9.4 months duration
37 person months effort
50% probability
500 FP Project
© Quantitative Software Management, Inc. #14
Ev a lu a te Pr o b a b ility o f C u rr e n t Es tim a te
L ife Du r at io n (M o n th s ) Ris k Pr o f ile<500 FP p rojec t>
0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 1 0 0
A s s uranc e Lev e l (% )
7
8
9
1 0
1 1
1 2
1 3
Life Duration (M
onths)
L ife Ef fo r t (PM ) Ris k Pr o file<500 FP p rojec t>
0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 1 0 0
A s s uranc e Lev e l (% )
0
2 0
4 0
6 0
8 0
1 0 0
Life Effort (P
M)
Life Dur ation (Months ) R is k P r ofi le - P robabi l ity de mo<5 0 0 FP pr oje ct>
A s s uranceLeve l(% )
15
10152025303540455055606570758085
Life Dur ation(Months )
7.267.908.258.488.668.828.969.099.219.339.459.579.699.819.94
10.0810.2410.42
Life Effort (P M) R is k P r ofile - P r obabi li ty de mo<5 0 0 FP pr oje ct>
A s s uranceLeve l(% )
15
10152025303540455055606570758085
Life Effor t(P M)
0.005.11
12.2016.9920.7924.0526.9829.6932.2734.7537.2039.6542.1344.7147.4250.3553.6157.41
P ro jec t: P robab i l i ty dem o
Likely outcomes 10.2 months schedule, 43 effort months
500 FP Project
© Quantitative Software Management, Inc. #15
S ta ffin g & P ro b a b ility An a ly s is
Avg S ta ff L i fe Cycle (pe o p le )< 15% s ize grow th>
1 2 3 4 5 6 7 8 9 10 11O ct'07
N ov D ec J an'08
F eb M ar A pr M ay J un J ul A ug S ep
0
2
4
6
8
Avg S
taff Life Cycle (people)
87654321
M iles tones 0 - C SR 1 - SR R 2 - HLDR 3 - LLDR 4 - C U T 5 - IC 6 - ST C 7 - U AT 8 - F C R 9 - 99R 10 - 99 .9R
M iles tones 0 - C SR 1 - SR R 2 - HLDR 3 - LLDR 4 - C U T 5 - IC 6 - ST C 7 - U AT 8 - F C R 9 - 99R 10 - 99 .9R
S OL U TI O N P AN E L - <1 5% size g ro wt h >
D u r ationE ffor tC o st
P ea k S ta ffM T T D
S tar t D a te
C & T7.335
60 3.76.5
1.6 8112 /28 /2 00 7
L if e C y cle1 0.24 6
7 87 .56 .5
1 .68 11 0/1 /2 00 7
M on th sP M$ (K )pe op leD a y s
P I =16 .5 M B I =3 .4 E ff F P =57 5
15% Growth (575 FP)
10.2 months duration
46 person months effort
© Quantitative Software Management, Inc. #16
E v a lu a te P r o b a b il i t y o f C u r r e n t E s t im a te
L if e Du r a t io n (M o n t h s ) R is k P r o f ile<1 5 % s iz e g r o w th >
0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 1 0 0
A s s u r a n c e L e v e l (% )
7
8
9
1 0
1 1
1 2
1 3
1 4
Life Duration (M
onths)
L if e Ef fo r t ( P M ) R is k P r o f ile<1 5 % s iz e g r o w th >
0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 1 0 0
A s s u r a n c e L e v e l (% )
0
2 0
4 0
6 0
8 0
1 0 0
1 2 0
Life Effort (P
M)
L i fe D u r ati on (M on th s ) R i s k P r o fi l e - P r o ba bi l i ty de m o< 1 5 % s i z e g r owth >
A s s u r a n c eL e ve l(% )
15
10152025303540455055606570758085
L i fe D u r ati on(M on th s )
7.838.518.889.129.329.489.639.779.91
10 .0 310 .1 610 .2 910 .4 110 .5 510 .6 910 .8 411 .0 011 .2 0
L ife Effo r t (P M ) R is k P r ofi l e - P r obabi l i ty de m o< 1 5 % s i z e g r owth >
A s s u r a n c eL e ve l(% )
15
10152025303540455055606570758085
L i fe Effor t(P M )
0.008.61
16 .7 622 .2 726 .6 430 .3 933 .7 636 .8 939 .8 542 .7 145 .5 248 .3 351 .1 954 .1 557 .2 860 .6 564 .4 068 .7 7
P ro j ec t: P ro b ab i l i ty d em o
15% Growth (575 FP)
Averages close to numbers predicted for effort and schedule growth (10.2 duration and 43 staff months of effort)
© Quantitative Software Management, Inc. #17
Staffing & ScheduleStaffing & ScheduleValidate Estimate with History
C&T Duration (Months) vs Effective SLOC
100 1,000Effective SLOC (thousands)
1
10
100
C&
T Duration (M
onths)
C&T Effort (PM) vs Effective SLOC
100 1,000Effective SLOC (thousands)
10
100
1,000
10,000
C&
T Effort (P
M)
Current Solution Logged Solutions Historical Projec ts QSM 2005 Business Av g. Line Sty le 1 Sigm a Line Sty leProjec t: Staffing & Schedule Demo
Schedule varies by a factor of 3.5 from -1σ to +1 σ
Effort varies by a factor of 8 from -1σ to +1 σ
What is “normal” variability?
© Quantitative Software Management, Inc. #18
How Should Project Effort Be ExpendedHow Should Project Effort Be ExpendedA Case StudyA Case Study
•• 838 projects that had data reported for Analysis/Design as well 838 projects that had data reported for Analysis/Design as well as Construction and Test phasesas Construction and Test phases
•• Average Effort applied to Analysis/Design = 20%Average Effort applied to Analysis/Design = 20%
•• 474 projects in the sample used <= 20% design effort474 projects in the sample used <= 20% design effort–– Average Analysis/Design Effort = 11%Average Analysis/Design Effort = 11%
•• 364 projects in the sample used > 20% design effort364 projects in the sample used > 20% design effort–– Average Analysis/Design Effort = 33%Average Analysis/Design Effort = 33%
•• Size profiles of samples very similarSize profiles of samples very similar
© Quantitative Software Management, Inc. #19
ObservationsObservations
•• Projects with <20% effort in Requirements and Projects with <20% effort in Requirements and Design Design –– Took 12% longer to completeTook 12% longer to complete
–– Averaged 5.6% more effort (median 24.4% greater)Averaged 5.6% more effort (median 24.4% greater)
–– Had an average staff 14.6% higherHad an average staff 14.6% higher
•• But these projects did excel at one thing:But these projects did excel at one thing:–– Found 63.7% more defects in systems testFound 63.7% more defects in systems test
–– Had 127% more defects in the first 12 months after deliveryHad 127% more defects in the first 12 months after delivery
© Quantitative Software Management, Inc. #20
Understanding TradeUnderstanding Trade--offsoffs
1 43 3
a b
a b= =
× ×where and
Size = Effort Time Productivity
Additional schedule has a much larger impact on a software project than increased effort
© Quantitative Software Management, Inc. #21
The Estimating ConundrumThe Estimating ConundrumSchedule / Effort TradeoffSchedule / Effort Tradeoff
Duration
Effo
rt
Impossible Zone
Impractical ZoneFeasible Solutions
Uncertainty about Size and Productivity creates uncertainty about the Duration-Effort curve
© Quantitative Software Management, Inc. #22
The Estimating ConundrumThe Estimating ConundrumSometimes no Solution WorksSometimes no Solution Works
Duration
Effo
rt
Impractical Zone
Impossible Zone
Feasible Solutions
© Quantitative Software Management, Inc. #23
The Estimating ConundrumThe Estimating ConundrumRelax the ScheduleRelax the Schedule
Duration
Effo
rt
Impossible Zone
Impractical ZoneFeasible Solutions
© Quantitative Software Management, Inc. #24
The Estimating ConundrumThe Estimating ConundrumIncrease EffortIncrease Effort
Duration
Effo
rt
Impossible Zone
Impractical ZoneFeasible Solutions
© Quantitative Software Management, Inc. #25
The Estimating ConundrumThe Estimating ConundrumReduce Size (Functionality)Reduce Size (Functionality)
Duration
Effo
rt
Impossible Zone
Impractical ZoneFeasible Solutions
© Quantitative Software Management, Inc. #26
The Estimating ConundrumThe Estimating ConundrumAssume Higher ProductivityAssume Higher Productivity
Duration
Effo
rt
Impossible Zone
Impractical ZoneFeasible Solutions
© Quantitative Software Management, Inc. #27
ConclusionsConclusions
•• Measurement is an integral part of managementMeasurement is an integral part of management
•• Information required to make precise estimates is Information required to make precise estimates is unavailableunavailable at project startat project start--upup–– Estimate uncertainty decreases rapidly with more informationEstimate uncertainty decreases rapidly with more information
•• Project estimates understate effort, schedule, & Project estimates understate effort, schedule, & sizesize–– Estimating based on a larger size or at a higher assurance levelEstimating based on a larger size or at a higher assurance level
can account for thiscan account for this
•• The tradeThe trade--off between schedule & cost/effort is off between schedule & cost/effort is nonnon--linearlinear
© Quantitative Software Management, Inc. #28
ConclusionsConclusions
•• Effort spent in Analysis & Design pays Effort spent in Analysis & Design pays bigbigdividendsdividends–– Reduces overall project effort (cost$$$$)Reduces overall project effort (cost$$$$)
–– Reduces overall project scheduleReduces overall project schedule
–– Improves project qualityImproves project quality
© Quantitative Software Management, Inc. #29
QuestionsQuestions??