Estimation, Planning & Control Processes Can Make the Difference Between Project success and FailureDan Galorath

galorath@galorath.com

Copyright 2011 Galorath Incorporated

Introduction

• Estimating is critical for all kinds of systems

• Yet many treat is as a second rate process

• Having a repeatable estimation process is critical to both estimating AND to successful projects

• Estimation and measurement go hand in hand

• While Galorath addresses Hardware, Software, IT , and manufacturing

• Context of this talk will be largely from software

© 2009 Copyright Galorath Incorporated 2

Delusions of Success: How Optimism Undermines Executives' Decisions (Source: Richard Hartley, HBR)

• Problem:

Humans seem hardwired to be optimists

• Optimism from cognitive biases & organizational pressures• Exaggerate talents & degree of control

• Attribute negative consequences to external factors

• Anchoring (relying too heavily on one piece of information) magnifies optimism

• Most pronounced for new initiativesSolution: Temper with “outside view”

Supplement traditional forecasting w/ statistical Parametrics

Don’t remove optimism, but balance optimism & realism

Example of Tempering: (Source How To Measure Anything)

• German Mark V Tanks

• Allies estimated production by analyzing serial numbers

• Used the captured tanks as a random sample and predicted probability of various production levels

An Estimate Defined

• An estimate is the most knowledgeable statement you can make at a particular point in time regarding:

• Effort / Cost

• Schedule

• Staffing

• Risk

• Reliability

• Estimates more precise with progress

• A WELL FORMED ESTIMATE IS A DISTRIBUTION

5

Software Is A Critical Component of Military & Aerospace Systems Failures Ariane 5

• Ariane 5 video

• Note: $500 Million payload

• Failure due to reused software from (Ariane 4)

• with incompatible assumptions for exception condition that was not required

ariane_5_explosion_video.mp4

“the culture within the Ariane program…” only addressed “random hardware failures… which can quite rationally be handled by a backup system”“the view had been taken that software should be considered correct until it is shown to be at fault”!

Why should we care: Software cost & schedule should be sufficient for successful missions

Software Is A Key Risk Item In Weapons Systems

• Navy Mobile User Objective Satellite Communication System delays to the Joint Tactical Radio System, a set of software-defined radios causes advanced MUOS capabilities to drastically underused… GAO

• GAO identified 42 programs at risk for cost & schedule 1. military requirements changes

2. software development challenges

3. workforce issues

• National Institute of Standards and Technology (NIST)

• Software defects cost nearly $60 Billion Annually

• 80% of development costs involve identifying and correcting defects

Software, not Hardware or technology readiness levels were called out

Software Is Getting More Complex Source: (Impact of Weapon System Complexity on Systems Acquisition by Robert A. Dietrick, Major, USAF)

Why should we care: Software is already difficult, costly, and error prone. More complexity means more problems

Software Is Providing an Increasing Percentage of Functionality

• Software development has been problematic for military aircraft developers

• Consuming about 40 percent of the USAF’s RDT&E budget

• Average software schedule overrun was 222 percent of the original estimate

• Overruns often justified by stating weapon systems performance requirements are evermore demanding

• And thus cause greater reliance on software

• Software provided about 10 percent of an F-4’s functionality in the 1960s but provides over 80 percent of an F-22’s

Why should we care: “unrealistic cost estimates lead to unrealistic budgets and un-executable programs” Meier Study… DAU

Software Failures Are Costly and Dangerous

• December 23, 2004, internal memo by Deputy Secretary of Defense called for the cancellation of the U.S. Air Force’s C-130J transport aircraft to cut its losses on the overpriced, unneeded, and problem-plagued C-130J. Estimated cancellation cost were $1.78 Billion – in 2004 dollars!

• The loss of a classified satellite after only 7 seconds on orbit prompted the review of software and processors that has caused the most recent delay and a potential $1 billion overrun in Lockheed Martin's Space-Based Infrared System (SBIRS). Strategic Command is worried about potential gaps in coverage for some mission areas in part because satellites are being delivered later than planned.

• Iraqi Scud missile slipped through Patriot missile defenses a year ago and hit U.S. Army barracks in Dhahran, Saudi Arabia, killing 28 servicemen.

• Cutter Consortium Software Project Survey:

• 62% overran original schedule by more than 50%

• 64% more than 50% over budget;

• 70% had critical product quality defects after release

• Standish Group CHAOS Report

• 46% challenged

• 19% failed

• 35% successful

~$875 billion spent on IT~$300 billion spent on IT projects ~$57 billion wasted annually

IT Has Similar Failures

Estimate Total Ownership Costs; 60+% Can Be Infrastructure & Services

• Software Development

• Software Maintenance

• IT Infrastructure

• IT Services

© 2009 Copyright Galorath Incorporated 12

Poor Estimates Effect on Projects• Inaccurate estimates can reduce project success:

• Poor implementations

• Critical processes don’t scale

• Emergency staffing

• Cost overruns caused by underestimating project needs

• Scope creep

• Forever changing project goals

• Frustration

• Customer dissatisfaction

• Cost overruns and missed schedules

• Project Failures

• Important project business decisions made early with minimum knowledge & maximum uncertainty

13

Why should we care: Poor estimates and plans are a root cause of program failure

Manual Estimates: Human Reasons For Error (Metrics Can Help)

• Manual Task estimates yield SIGNIFICANT error

• Desire for “credibility” motivates overestimate behavior (80% probability?)

• So must spend all the time to be “reliable”

• Better approach force 50% probability & have “buffer” for overruns

• Technical pride sometimes causes underestimates

14

SEER: Software Analysis Tools A Complete Software Project Management Solution

Kno

wle

dge

&E

xper

ienc

e

Baseline(s) &

Final Actuals

To-date

Actuals

Measure&

Analyze

Monitor&

Control

Baseline

Estimate&

Plan

A Foundation of Risk Management

Quantitative Project Management

15

16

10 Step Software Estimation Process:Consistent Processes = Reliable Estimates

1. Establish Estimate Scope

2. Establish Technical Baseline, Ground Rules, Assumptions

3. Collect Data

4. Estimate and Validate Software Size

5. Prepare Baseline Estimates

7. Quantify Risks and Risk Analysis

6. Review, Verify and Validate Estimate

8. Generate a Project Plan

9. Document Estimate and Lessons Learned

10. Track Project Throughout Development

17

This chapter discusses a 1972 GAO report on cost estimating• We reported that cost estimates were understated and causing unexpected cost growth

• Many of the factors causing this problem are still relevant today We also discuss a 12 step process for producing high quality cost estimates

GAO Publication: Characteristics of credible cost estimates and a reliable process for creating them

18

GAO Publication: Why cost estimates are required for government programs and challenges developing credible results

• Introduces why cost estimates are required for government programs

• Developing annual budgets, supporting management decisions about which program to fund, and evaluating resource requirements at key decision points

• Discusses various challenges associated with developing credible results

© 2010 Galorath Incorporated

Contractors At Least Level 3 Would Be Acceptable

Why should we care? Maturity is related to estimate viability… Better estimation process more likely to

be successful in execution

Balancing Resources & Schedule Is A Science

For a given Size, Complexity and Technology

Minimum Time

Optimal Effort(Lower Effort for Longer Schedule)

Calendar Time

Eff

ort

Mon

ths

Effort Increasedue to Longer Schedule

0 4 8 12 16 20

Schedule ProbabilityExample Application 1

Probability

Time (calendar months)

1%10%20%30%40%50%60%70%80%90%99%

Understand Project Risks Include Them In Planning Decisions (Example SEER-SEM Outputs)

0 1800 3600 5400 7200 9000

Effort (person-hours)

1%10%20%30%40%50%60%70%80%90%99%

Effort ProbabilityExample Application 1

Probability

0 12 24 36 48 60

Defects ProbabilityExample Application 1Probability

Defects (count)

1%10%20%30%40%50%60%70%80%90%99%

21

Understanding & Tracking Defects, Growth And Other Metrics

Heath and Status Indicator shows status

and trends from the previous snapshot

•Including Size Growth and Defect Discovery/Removal

Rate

•User defined control limits to control the transition

between red-yellowyellow-green

Track defect

discovery and

removal rates

against expected

rates

Increased defect

reporting rate shows a

worsening trend

Track software

size growth 22

Considering Maintenance During Planning Can Yield More Successful Long Term Results

Staff Vs Maintenance Rigor

0500

100015002000250030003500

1 7 13 19 25 31 37 43 49 55 61 67 73 79 85

Time

staf

f hou

rs p

er m

onth

develop

Rigor vhi+

Rigor nom

Rigor vlo

23

Software & IT Systems Are About Business Value

Data Doesn’t Have To Be Perfect To Be Useful: But Is Has To Be Viable

• 80 Calories per serving

• 2.5 Servings per can

• 4 Ounces, Condensed, 8 Ounces With Water

You have an estimate …Now what?

26

• Correlation does not imply causation• Just because two data points may sit side by side

doesn’t mean they are the same or will have the same outcome

• Casual analysis is a recognized error in medicine

Perhaps ???

The Error of Causal AnalysisCreating a False Association

Tumor Can Cause Headache

Headache doesn’t mean a tumor

27

Use Historical Measurement to evaluate your estimate!

It’s easy to dig deeper and deeper to justify an estimate!28

29

NormalizationAdjustments for homogeneity

• Time phased data can be particularly difficult to handle

• As spent dollars are influenced by inflation• Requires knowledge of time period involved

• May require converting to “fixed dollar” units

• Process data• Need to know specific sub-phase of the process

• May require allocation to sub-phase

30

Normalization: What To Look For In An Actual

Phase : All activities may not be included.

System Concept missing

Integration missing

Phases

System Concepts System Req & Design

System Req Analysis Preliminary Design

Detailed Design Code / Unit Testing

Software Test System Integration / OT&E

31

Normalization: What To Look For In An Actual: Labor

Labor : All categories may not be included.

Quality Assurance missing

Configuration Mgt missing

Labor Categories

Management System Engineering

Design Coding

Data Preparation Test

Configuration Mgmt. Quality Assurance

Estimation Best Practices

• Decide Why You Want An Estimate

• Map Estimation Goals To Estimate Process Maturity & Develop Plan To Achieve The Maturity

• Have A Documented, Repeatable Estimation Process

• Make The Estimating Process As Simple As Possible; But No Simpler

• Be Proactive: The Process Is Important, The Tools Go Along With The Process 

• Get Buy-in From Program Managers

• Hold People Accountable: Center Of Excellence Can Prepare Estimate But Program Managers Must Own Them

• Tie The Estimate To The Plan

© 2009 Copyright Galorath Incorporated 32

Estimation Best Practices 2

• Evaluate Total Ownership Cost; Not Just Development

• Estimate A Range And Pick A Point For The Plan

• Re-estimate The Program When It Changes

• Avoid Death Marches: Programs With Unachievable Schedules Are Likely To Fail And Drain Morale

• Keep A History: Start An Enterprise Database NOW…

• Business Case: Evaluate ROI In Addition To Costs

• Convert Expert Spreadsheets Into A Common Language

© 2009 Copyright Galorath Incorporated 33

Estimation Best Practices 3

• Track Progress Vs. Estimate Throughout The Life Cycle

• Estimate Schedule As Well As Effort (Cost) For Complete Picture

• Tie The Business Case Into The Estimating Process

• Attack Non-productive Rework As Part Of The Process

© 2009 Copyright Galorath Incorporated 34

Estimation Best Practices 4

• Have clear definitions: What does “complete” mean. What activities are included and excluded (E.g. development only or total ownership; help desk included or excluded, etc.) Which labor categories are included and excluded in the estimate (e.g. are managers included? Help desk? Etc.)

• Measure what you care about

• Estimating & tracking rework can help control costs

• Don’t ignore IT infrastructure and IT services costs

• Tracking defect sources can go along with the process

© 2009 Copyright Galorath Incorporated 35

Additional Information

• www.galorath.com

• Dan on estimating BLOG: www.galorath.com/wp

• Email: galorath@galorath.com

© 2009 Copyright Galorath Incorporated 36

Economics Is a Key Driver In Software (Source: Royce)

© 2009 Copyright Galorath Incorporated 37