1Oct 2016

Cost Risk/Uncertainty Analysis Overview

Learning Objectives

• Terminal Learning Objective: Explain the differences between cost risk analysis and uncertainty analysis as applied to major weapon systems

• Enabling Learning Objectives:– Categorize the various methods used to perform cost risk & uncertainty analyses

– Classify the types of risk faced by defense acquisition programs.

– Illustrate the difference between risk and uncertainty

– Explain the Monte Carlo Simulation process used in cost risk analyses

2

General Cost Risk Modeling Process

3

PointEstimate

Statistical UncertaintyDistributions

Program RiskRegister

Uncertainties

Framing AssumptionsUncertainties

RunSimulation(s)

ApplyCorrelation

ApplyAny OtherInfluences

Interpret Results

Allocate andPhase Risk

Dollars

ReportResults

Consider other possible risks

Document as you go!

A Typical Program Cost Estimate

• The standard product of a cost estimating exercise is a single number – the point estimate – an anchor point

• We know that the estimate is wrong – we hope the actual cost will be close to the estimate

• We are not comfortable presenting the point estimate• many assumptions that may or may not come

true• many guesses were required to generate the

point estimate• how do we express the uncertainty in the

estimate?

4

Risk vs Uncertainty

Risk: the probability of a loss or injury

Opportunity: a favorable event or outcome

Uncertainty: the indefiniteness about the outcome of a situation

5

The goal of the simulation model is to combine all the sources of cost uncertainty in order to estimate the risk of exceeding a given budget.

Source: The Joint Agency Cost Schedule Risk & Uncertainty Handbook (CSRUH), p. 3

Typical Questions

• What distribution best fits our total cost data?

• What is the probability costs will exceed $1B?

• What is probability of a 30% or greater cost overrun?

• If X~Triang(15,35,85), what is P(X≥60)?

• If my funding is $150M, what is probability my program will exceed this funding level?

• What funding is required at the 60% confidence level?

6

Cost Risk & Uncertainty Analysis (CRUA)

• CRUA provides insights into these questions

• CRUA is a process of quantifying the cost impacts of uncertainties associated with a systems technical definition, cost estimating methodology, requirements, threat and schedule

7

Elements of Risk/Uncertainty in a Cost Estimate

• Technical Risk/Uncertainty– Risk/uncertainty due to inability to conquer technology problems

posed by the intended design in the current CARD or system specs

• Schedule Risk/Uncertainty*– Technical and other risks impact the schedule

• Concurrent development

• Test failures

• Delayed Milestone approval

• Optimistic task durations

• Requirements Risk/Uncertainty– Variations caused by unforeseen design shift from the current CARD

or system specs

• Programmatic Risk/Uncertainty– Beyond control of PM office; program decisions made at higher levels

of authority; includes budgeting decisions

8*Adapted from GAO Cost Estimating & Assessment Guide (2009)

Examples

• Threat Risk– Risk due to an unrevealed threat

– The problem changed (which changes the solution, design, & cost)

• Cost Estimating Risks– Variations despite a fixed configuration baseline

– Inaccurate, imprecise, or incomplete data leading to biased or imprecise estimates

– CER model misspecification

– Mathematical/statistical errors

• Business or Economic– Variations caused by changing assumptions

• Other?Cost Risk = ∑ All the Above Risks 9

Elements of Risk/Uncertainty in a Cost Estimate

• Parametric CERs including factors and cost improvement curve (CIC) equations

• CER inputs, complexity factors for analogies, engineering judgment

• Any other cost drivers (man-hours, head counts, rates, ratios, overhead, fee, etc.)

• The planned schedule (durations)

• Risk register events, both probability of occurrence and the consequence

• Framing Assumptions

10

Sources of Uncertaintyto be Captured in CRUA

See CSRUH p. 3 for more on sources of uncertainty

Types of CRUA

• Historically-based methods– Uses mean growth or distribution of growth from a set of

analogous programs

– Example: “On average, this type of program has experienced 17% cost growth. Therefore, we will increase the point estimates by 17%.”

– Growth factors should be adjusted based on extent of risk present in program being estimated vs average program from analogous data set

• Expert Opinion-based methods– Ask experts to assess the uncertainty around cost estimates, by

cost element

– Have them “score” a program (and other programs) in terms of the various risk elements; map scores to dollars

11

Types of CRUA

• Input-based methods– Assess uncertainty around inputs to the cost model, as well as the CER

equations themselves (where applicable)

– Most simulations use Input-Based methods (e.g. Monte Carlo)

– Symmetric approximation (a Method of Moments technique) is an input-based method

• Output-based methods– Apply uncertainty directly to estimate outputs (e.g., WBS elements) by

assigning distributions and simulating

• Scenario-based methods– From the point estimate, define a “protect scenario” that captures the

impacts of the major known risks to the program

– “Protect scenario” is combination of multiple sensitivities

– Determine cost of protect scenario; delta between this and point estimate is risk dollars

– May be used to capture potential impacts of framing assumptions

12See CSRUH p. 5 for more on types of CRUA, p. 80 for discussion on Method of Moments

Cost as a Probability Distribution (PD)

• Cost is an uncertain quantity • Cost is highly sensitive to conditions and assumptions

that change across a systems life• Sensitivity analysis

– Change in cost…varying conditions…isolates cost drivers– A deterministic process…defined by set of single variable

changes– Does not offer insights into simultaneous changes– Does not result in a distribution representing potential range of

costs

• A PD is a way to address many questions– A mathematical rule associating a probability to each outcome– Two types are PDF and CDF

13

Cost as a Probability Distribution (PD)

• Basis – if, individual cost elements are random variables and their distributions determined, then system costs can be expressed as PD

• Problem – how to determine PDs

• Problem – how to combine many individual cost elements and their uncertainty into a total estimate of cost representing all the inherent uncertainty

• Solution – Symmetric Approximation (Summation of Moments) and Monte Carlo Simulation

14

Cost Risk/Uncertainty Analysis

• Using Input-based Monte Carlo Simulation

– “….offers the user a powerful and precise method of assimilating the various uncertainties of a problem and producing a realistic appreciation of the problem’s uncertainty.” (David Vose, Quantitative Risk Analysis, p1)

– Accounts for values across a range and weights them by probability of occurrence

– Models each uncertain variable by a probability distribution function…assigns probabilities

– Objective – calculates the combined impact of the model’s various risks in order to determine a PD of possible model outcomes

15

• A PDF characterizes the probability associated with all possible outcomes of a random variable

• If we view cost as a random variable, then a PDF is an effective way to depict the possible outcomes

A Cost Probability Density Function (PDF)

0

1

2

3

4

5

6

7

8

1.11

1.13

1.15

1.17

1.19

1.21

1.23

1.25

1.27

1.29

1.31

1.33

1.35

1.37

1.39

1.41

1.43

1.45

1.47

1.49

1.51

1.53

1.55

1.57

1.59

Cost

Once PDF determined, can make probability statements with respect to costs.P

roba

bili

ty

16

The Total Cost PDF

17

AREA = 10%There is a 10% chance

that cost will exceed $1.40MOR

There is a 90% chance that the program will not exceed a $1.40M budget

AREA = 50%There is a 50% probability

that cost will be less than $1.32MP

roba

bili

ty

18

0

1

2

3

4

5

6

7

8

1.11

1.13

1.15

1.17

1.19

1.21

1.23

1.25

1.27

1.29

1.31

1.33

1.35

1.37

1.39

1.41

1.43

1.45

1.47

1.49

1.51

1.53

1.55

1.57

1.59

Cost

Pro

babi

lity

Probability Density Function

Cost

The Total Cost PDF

Using PDFs

• Identify a PDF for the uncertain inputs in the cost estimate

• Most commonly used in CRAs are:

° Lognormal

° Triangular

° Beta (PERT Beta)

° Normal

° Uniform

° Discrete

• Identify parameters (e.g., mean, SD, high, low, ML, discrete values)

° Historical data

° Expert opinion

° Translate identified risks into possible parameter values

• Using Monte Carlo Simulation, combine the input PDFs into the potential total cost PDF

19

Adapted from CSRUH, Table 2-2

Distribution Typical Application Parameters

LognormalDefault when no better info. Probability skewed 

right.  Power OLS CER uncertainty

Mean & Standard Deviation.  Some tools have a 3rd parameter: "Location“.  By default, it is zero.  Used to "shift" the lognormal left or right (even 

into the negative region).

TriangularExpert opinion. Finite min/max. Probability reduces towards endpoints. Skew possible. Labor rates, labor 

rate adjustments, factor methodsLow, mode and high

BetaPertLike triangular, but mode is 4 times more important 

than min or maxLow, mode and high

NormalEqual chance high/low. Unbounded in either 

direction. Linear OLS CER uncertainty.Mean & Standard Deviation

UniformEqual chance over uncertainty range.              

Finite min/max.Low and high (some tools require min and mix)

Empirical Fit Unable to fit a distribution to the dataEntire source data and estimated probability for 

each data point

Note:  Low/high are defined with an associated percentile

Min/max are the absolute lower/upper bounds (also known as the 0/100)

Recommended Uncertainty Distributions

20

Lognormal Distribution• Often used – intuitively appealing

• Right skewed; bounded by zero

• Mode, median and mean all different

• Natural result of non-linear CERs

• Looks Normal at CVs less than 25%

21Cost ($K)

Pro

bab

ilit

y

• Parameters are µ and σ

Triangular Distribution• Simple to apply, understand and communicate

° parameters are high, low, and most likely

• Use when you are confident of the bounds

22

Weight (lbs)LowMost Likely

High

Pro

bab

ilit

y

PERT Beta Distribution

• Flexible distribution -- can take many shapes

• Easy to specify parameters° Low, Mode (Most Likely) and High are specified

° Costs easily fitted – does not go negative.

23

Cost ($K)

PERT: Project Evaluation & Review Techniques

Pro

bab

ilit

y

Min = 0, ML = 300, Max = 1000

Min = 0, ML = 500, Max = 1000

Min = 0, ML = 700, Max = 1000

0 200 400 600 800 1000

Normal Distribution• Symmetric

° Equal probability the input will be either higher or lower than the most likely

° Use cautiously for costs

• Parameters are µ and s

24

Pro

bab

ilit

y

Cost ($K)

Low High

Uniform Distribution• All outcomes between high and low are equally likely

° There is no most likely value

• Use when you have no idea about the relative likelihood of possible outcomes

25Weight (lbs)

Pro

bab

ilit

y

Discrete Distribution

• Outcomes are finite

• Probability assigned for each possible outcome

• Several forms of discrete distributions are useful

26

Pro

bab

ilit

y

Outcome

Outcome Probability2 0.0278 1/363 0.0556 2/364 0.0833 3/365 0.1111 4/366 0.1389 5/367 0.1667 6/368 0.1389 5/369 0.1111 4/36

10 0.0833 3/3611 0.0556 2/3612 0.0278 1/36

1.0000 36/36

Monte Carlo Simulation

• Process– Draw one observation from each input distribution

– Calculate a total cost with the set of drawn input values

° Store each total cost value as one data point

– Repeat many times° How many iterations are sufficient?

° Will more iterations “fill out” the total cost distribution?

27

Monte Carlo Simulation

141.2 hrs * $62.41/hr = $8,812

125 lbs * $4.50/lb = $562

Obs. Total Cost

Results (data)

Multiply

divide

Add

141.2 hrs * $????/hr = $????

??? lbs * $4.50/lb = $???

Cost Model and Estimating

Relationships

Inputs

1 $24,542

2 $30,218

3 $26,871

1000 $25,611

$62.41/Hr

125 lbs

8.3%

28

0

20

40

60

80

100

120

23 23.5 24 24.5 25 25.5 26 26.5 27 27.5 28 28.5 29 29.5 30 30.50

200

400

600

800

1000

1200

23 23.5 24 24.5 25 25.5 26 26.5 27 27.5 28 28.5 29 29.5 30 30.5

Obs. Total Cost

1 $24,542

2 $30,218

3 $26,871

4 $22,988

1000 $25,611

Simulation Results

Cost $K Cost $K

Cu

mu

lati

ve F

req

uen

cy

Fre

qu

ency

Cost Model and Estimating

Relationships

29

30

Cu

mu

lati

ve P

rob

abil

ity

1.0

0.8

0.6

0.4

0.2

0

Cost $K

Cum Frequency Diagram to a Cum Density Function (CDF)

Risk (Contingency) Dollars

• The point estimate will usually be somewhere between the 15th and 50th percentile*° Depends on skewness of the PDFs

• Decision-makers may budget at a higher level

• Risk (Contingency) is the difference between the point estimate and the budget

° Where do you put that money?

31*See Smart, 2015, p.201 and CSRUH p. 60

General Cost Risk Modeling Process

32

PointEstimate

Statistical UncertaintyDistributions

Program RiskRegister

Uncertainties

Framing AssumptionsUncertainties

RunSimulation(s)

ApplyCorrelation

ApplyAny OtherInfluences

Interpret Results

Allocate andPhase Risk

Dollars

ReportResults

Consider other possible risks

Document all along!

Summary

• Several methods available for conducting a CRUA– This course will focus on the Input-based method

using Monte Carlo simulation

• Requires understanding of simulation concepts and probability distributions

• A good CRUA looks at all facets of uncertainty in the estimate……..– It should have a strategy

– It should be able communicate the results

33

References

34

Department of Defense. 2014. Joint Agency Cost Schedule Risk and Uncertainty Handbook(CSRUH). http://cade.osd.mil/cade/CSRUH.aspx

Smart, Christian. 2013 “Default Correlation for Cost Risk Analysis.” Missile Defense Agency, Paperpresented at the ICEAA Professional Development and Training Workshop, New Orleans, LA. June18-21.

Smart, Christian, 2015 “Covered with Oil: Incorporating Realism in Cost Risk Analysis”, Journal of Cost Analysis and Parametrics, 8:3, 186-205, DOI: 10.1080/1941658X.2015.1096220

Capen, E.C. 1976. “The Difficulty in Assessing Uncertainty.” Society of Petroleum Engineers.28(8):843-850. doi: 10.2118/5579-PA.

Government Accountability Office. 2009. Cost Estimating and Assessment Guide: Best Practices forDeveloping and Managing Capital program Costs. http://www.gao.gov/products/GAO-09-3SP .

Department of Defense. 2008. Risk Management Guide (RMG) For DoD Acquisition Sixth Edition (Ver1.0). http://www.everyspec.com/DoD/DoD-PUBLICATIONS/RISK_MGMT_GUIDE_FOR_DOD_ACQUISTION_2605/

Department of Defense. 2015. Risk, Issue, and Opportunity Management Guide for Defense Acquisition Programs. www.acq.osd.mil/se

Book, Stephen A. 2007. “Allocating “Risk Dollars” Back to Individual Cost Elements.” Paper presented at 40th Annual DoD Cost Analysis Symposium, Williamsburg, VA, February 15-18.

Book, Stephen A. 1999. “Why Correlation Matters in Cost Estimating.” Paper presented at 32ndAnnual DoD Cost Analysis Symposium, Leesburg, VA, February 2-5.

BACKUP

35

• STEP 1: Determine program cost drivers and associated risks

• STEP 2: Develop probability distributions to model uncertainty

• STEP 3: Account for correlation between cost elements

• STEP 4: Perform uncertainty analysis (e.g. with Monte Carlo simulation)

• STEP 5: Identify the probability associated with the point estimate

• STEP 6: From confidence level, determine risk dollars

• STEP 7: Allocate, phase, and convert the risk-adjusted estimate to then-year dollars and identify high risk elements

GAO Cost Risk Modeling Process

36Source: GAO Cost Estimating and Assessment Guide (2009), Chapter 14