Amer v2

DEVELOPMENT OF UNCERTAINTY FACTORS FROM

HISTORICAL NASA PROJECTS

Dr. Tahani Amer, OE/IPAO NASA-HQDr. Will Jarvis, OE/IPAO NASA-HQ

Dr. Ariel Pinto, Old Dominion University

NASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage.

Project Background

• Uncertainty analysis should be performed to capture the program risks.

• NASA has been using available uncertainty factors from Aerospace, Air Force, and Booz Allen Hamilton.

• NASA has no insight into the development of these factors, which can lead to unrealistic risks in most NASA projects.

• SRB analysis requires cost growth factors that are transparent and based on recent NASA experience.

• Better estimation and prediction of missions cost is needed early on.

NASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 2

Data Collection Methodology

• NASA Fiscal Year Budget Estimates

• Cost Analysis Data Requirement (CADRe)

• Government Accountability Office Reports

• NASA Personal- IPAO/CAD/SMD


Data Collection Model

GAO Report

Budget Report

Analysis- Recheck

Analysis- Recheck

Assess Assess Assess

CA

DR

e R

epo

rt Data S

et


5

FY 2009 BudgetFebruary 4, 2008

Amer DENG

Source of Data


Data Sampling Criteria

• Difficulty in collecting accurate data – limited dataset

• Criteria– Science Mission Projects– Developmental Phase– Only cost data– Use CARDe data as final for completed Missions– For active missions: used in search of credible earliest available

and latest available estimates. Errors probably exist because some projects are still in development and continue to evolve


Summary of NASA Missions Investigated

Completed Missions Active Missions

NEARLUNAR PROSPECTOR GENESISMESSENGERMARS PATHFINDER STARDUST CONTOURDEEP IMPACT MGSMCO/MPL

MERMRO FAST SWAS TRACE WIRE ACE FUSE IMAGEMAP

HESSI GALEX SWIFT GRACE CLOUDSAT CALIPSODS-1EO-1SIRTFSTEREO

EOS-Aqua EOS-Aura LANDSAT-7TRMM TIMEDGRAVITY PROBE B THEMISHETE-II SORCEICESAT

AIM DAWN PHOENIX GLAST KEPLER SDOWISENEW HORIZONS LROOCO

JUNO AQUARIUS LDCMNPP GPM MMS JWSTMSL GLORYGRAIL


Initial & Final Cost for 50 Completed Projects and Cost Growth Percentage


60 NASA Missions - Historical Data


Mission Cost Growth Histogram


NASA & DOD Distribution

0

2

4

6

8

10

12

14

Num

ber o

f Pro

ject

-21-0% 0 -10% 10-20% 20-30% 30-40% 40-50% 50-60% 60-70% >70%

Permission from McCrillis, DOD


60 NASA Missions

0

2

4

6

8

10

12

14

Num

ber o

f Pro

ject

- 21-0% 0 -10% 10-20% 20-30% 30-40% 40-50% 50-60% 60-70% >70%


Reasons for Cost Growth

• The major categories for cost growth:

– Overly optimistic and unrealistic initial cost estimates– Project instability and funding issues– Problems with development of instruments and other spacecraft

technology– Launch service issues– “Stuff” Happens


Level Optimistic Most Likely Pessimistic

Low 0.95 1 1.1

Low Plus 0.96 1 1.23

Moderate 0.97 1 1.36

Moderate Plus 0.98 1 1.49

High 0.98 1 1.61

High Plus 0.99 1 1.74

Very High 1 1 1.87

Very High Plus 1 1 2

Aerospace Air Force

Booz Allen Hamilton

Uncertainly Factors from non-NASA Missions


NASA Data & Air Force UF


NASA Data & Aerospace UF


NASA Data & BAH UF


Development of Distribution-54


Development of Distribution-44

Lognormal

Cost Growth (%)1501401301201101009080706050403020100

PD

F

0.12

0.11

0.1

0.09

0.08

0.07

0.06

0.05

0.04

0.03

0.02

0.01

0

F(X) =

µ= 3.03σ= 1.08


More Test Runs

• All data including Project A & B (A- Data), 54 missions.• All data excluding Project A & B (B- Data), 52 missions.• All completed missions (C-Data), 44 missions.• All data less than 100% cost growth (D- Data).

Type of Data µ =Mean σ= Std Dev.

A - Data 3.01

20.3 %

1.18

B - Data 2.92=

18.5 %

1.11

C - Data 3.03

Mean=37.08

Median= 20.7

Std. Dev.= 55.14

1.08

D-Data 2.85

17.3%

1.07

October 7, 2011 Lognormal

Cost Growth (%)1501401301201101009080706050403020100

PD

F

0.12

0.11

0.1

0.09

0.08

0.07

0.06

0.05

0.04

0.03

0.02

0.01

0


Selected Distribution

• The lognormal distribution is the most suitable distribution that the cost estimator can use to perform an uncertainty analysis for the NASA data.

• The quality of the fit of two tests: – The Kolmogorov-test – The Anderson-Darling test

Lognormal (1.08; 3.03)

Cost Growth (%)100500

PD

F

0.032

0.028

0.024

0.02

0.016

0.012

0.008

0.004

0


Recommended UFs

Level of RiskUncertainty Factors

No- Risk Adjusted Conservative Aggressive

Moderate (10-30%) 1 1.1 1.3

High (30-75%) 1 1.3 1.75

Very High (>75%) 1 1.75 2.5


UFs w/Conservative High Risk Level


UFs w/Aggressive Moderate Risk Level


Guideline to UFs

• Cost Analyst Judgment

• Mission Complexity: Number of Instruments, Technical Development

• Heritage Level

• Partners Performance

• Mission Visibility: International & Commercial

• Similarity of Missions


Comparison of UFs


UFs ------Test Case

– Project A---• Development Cost as of 2006 ~ $969 M• Development Cost as of 2011 ~ $1.8 B• Apply NUFs:

– Make the following assumptions:» High Risk Mission (10 instruments, new Technology, Planetary)» Aggressive Risk » Select a range from NUFs, Table 20, 1.75-2.5

• Expected development estimate range: 1.69- $2.42 B• Other factors estimates:

– AF (1.61- 1.74) $1.56B- $1.69 B, which is lower than the current actual development cost.

– Aerospace (2.05-2.5) $1.99- $2.52 B, which is much higher then expected estimate from the low range. Note: LCC is @$2.5B.

– BAH (1.9-2.10) $1.84- $2.03 B, which is might be closer to the expected development cost . Note: see section 3.3 for these factors interpolation from BAH table.


UFs Test Case


.

Current EST.

970 1.2 1.4 1.8 2.2 2.6 2.8

UF

AF

AS

BAH

$1.69- $2.42B

$1.56B- $1.69B

$1.99- $2.52B

$1.84- $2.03B

Conclusions

• This research provides an important contribution to the discipline of cost estimating.

• It has developed a solid database of actual cost growth history and adds some statistical rigor to the derivation of cost growth factors based on the data.

• Thoughtful execution and effective monitoring of the NUFs will improve the cost estimating.

• Additionally, it is expected that this work will be referenced often for independent cost estimates, correction of advocacy-bias in project-generated estimates, and other programmatic work.


Point Taken

• The greatest benefit of this research is providing a proactive approach to make a quick cost estimating as in back of envelope by:

– Program Managers– Decision Makers– Congressional Staffer– Cost analysts– Awareness and reference to Standard Review Board


Future Work

• Increase the number of historical missions captured in the database.

• Validate UFs with more available actual cost data.

• Test the UFs for missions in various life-cycle phases.

• Develop a more adaptable tool for NUFs.

• Accept the UFs by the cost analysis’ community for implementation.

• Develop Agency standard.


Backup


Amer v2

Technology

Transcript of Amer v2