Post on 11-Dec-2015
28 Aug 2012 PRT-148
Los Angeles Washington, D.C. Boston Chantilly Huntsville Dayton Santa Barbara
Albuquerque Colorado Springs Goddard Space Flight Center Johnson Space Center Ogden Patuxent River Washington Navy Yard
Ft. Meade Ft. Monmouth Dahlgren Quantico Cleveland Montgomery Silver Spring San Diego Tampa Tacoma
Aberdeen Oklahoma City Eglin AFB San Antonio New Orleans Denver Vandenberg AFB
Joint Analysis of Cost and Schedule (JACS) Joint Analysis of Cost and Schedule (JACS)
Australian Department of DefenceAustralian Department of Defence2nd Cost Estimation Conference 2nd Cost Estimation Conference
29 - 30 October 2012 29 - 30 October 2012
Alfred Smith, CCEAAlfred Smith, CCEAJennifer Kirchhoffer, CCEA Jennifer Kirchhoffer, CCEA
Joint Analysis of Cost and Schedule (JACS) Joint Analysis of Cost and Schedule (JACS)
Australian Department of DefenceAustralian Department of Defence2nd Cost Estimation Conference 2nd Cost Estimation Conference
29 - 30 October 2012 29 - 30 October 2012
Alfred Smith, CCEAAlfred Smith, CCEAJennifer Kirchhoffer, CCEA Jennifer Kirchhoffer, CCEA
28 Aug 2012 PRT-148 Approved for Public Release 2 of 35
What is JACS?
Overview of the JACS modeling process
Key reports from a well constructed JACS model
Concluding remarks
AgendaAgenda
28 Aug 2012 PRT-148 Approved for Public Release 3 of 35
Cost, schedule and risk assessments traditionally have been performed by separate teams of professionals
In recent years, it has become more common for the cost analyst to report a “risk adjusted” result as a budget recommendation rather than a point estimate
However, it appears that cost uncertainty models routinely: try to force a 70 or 80% cost result into the point estimate schedule ignore risk management team statements like “High probability this
event will occur and if it does, the consequence will be severe”
What is Joint Analysis of Cost and Schedule (JACS)?
What is Joint Analysis of Cost and Schedule (JACS)?
Joint Analysis of Cost and Schedule is a disciplined, systematic and repeatable process to integrate three critical pieces of information: Cost, Schedule, Risk
28 Aug 2012 PRT-148 Approved for Public Release 4 of 35
Current Approach to Model Cost Estimating UncertaintyCurrent Approach to Model Cost Estimating Uncertainty
Cost
Input, e.g., weight
Cost = a + bxc
Historical data points
Sources of Uncertainty:•Cost estimating method•Cost method inputs
Focus is on estimating total cost uncertainty with limited influence from duration uncertainty or potential events that may influence cost/schedule.
28 Aug 2012 PRT-148 Approved for Public Release 5 of 35
Capture schedule uncertainty on Time-Dependent Costs
Inclusion of Discrete Risks (5x5’s)
Evolving Trends in Uncertainty and Risk Analysis
Evolving Trends in Uncertainty and Risk Analysis
Time-Dependent (TD)[Level of Effort - LOE]
Risk 1
Risk 2
Risk n
.
.
.
Translate 5x5 into probability of occurrence times uncertain consequence which can impact cost and/or duration of one or more tasks
1
n
2
28 Aug 2012 PRT-148 Approved for Public Release 6 of 35
Project Start
Project End
Task Duration
Integrated Risk & Uncertainty Landscape – the JACS Paradigm
Integrated Risk & Uncertainty Landscape – the JACS Paradigm
TD = Time-Dependent Cost, e.g. ‘marching army’ cost
TI = Time-Independent Cost, e.g., Materials
Duration Uncertainty
Burn Rate Uncertainty
TD $ = Segment Duration X Burn Rate
TD $ = Segment Duration X Burn Rate
Probability of Occurrence
Risk Register
TI $
TI $
TI $
TI $
TI $TI $ Uncertainty
TI $
28 Aug 2012 PRT-148 Approved for Public Release 7 of 35
What is JACS?
Overview of the JACS modeling process
Key reports from a well constructed JACS model
Concluding remarks
AgendaAgenda
28 Aug 2012 PRT-148 Approved for Public Release 8 of 35
The JACS Process:Develop the Analysis Schedule
The JACS Process:Develop the Analysis Schedule
Risk
Sched
Cost
CollectSched
Data
Create AnalysisSchedule
ValidateBefore
Continuing
28 Aug 2012 PRT-148 Approved for Public Release 9 of 35
Joint analysis of cost and schedule begins with a model of the schedule logic Serves as the backbone for the analysis Cost, risks and uncertainty are mapped into the logic to assess
impacts
Project/program integrated master schedules (IMS) are unsuitable for this role They are generally too big, complex and too detailed Logic common in an IMS can be a problem for a JACS analysis (e.g.,
constraints)
A JACS appropriate schedule must be created from available data (including the program IMS) Typically referred to as an “analysis schedule”
The Need for an Analysis ScheduleThe Need for an Analysis Schedule
28 Aug 2012 PRT-148 Approved for Public Release 10 of 35
√ Captures the major work-flows of the project IMS
√ Provides insight into major cross-dependencies within or across management responsibility boundaries
√ Creates a solid framework to capture cost / schedule uncertainties and discrete risk events
√ Structured around management/ budget responsibility
√ Allows mapping of budgeted work effort to schedule scope
√ Aligns with cost/budget data
√ Identifies key tasks that support major deliverables/ tracking items
√ Detailed IMS step by step work items and task flows are combined while maintaining critical path logic
√ Has traceability and transparency to the more detailed IMS
Attributes of an Analysis ScheduleAttributes of an Analysis Schedule
28 Aug 2012 PRT-148 Approved for Public Release 11 of 35
The JACS Process:Map Costs to Schedule Tasks
The JACS Process:Map Costs to Schedule Tasks
Risk
Sched
Cost
CollectSched Data
Create AnalysisSchedule
UpdateAnalysisSchedule
Collect Cost Data
Identifyas
TD or TI
Map toSched
Activities
28 Aug 2012 PRT-148 Approved for Public Release 12 of 35
Easier to map cost model results to a schedule model rather than replicating schedule network logic in a cost model
Schedule models are generally populated with throughputs, that is they don’t allow equations to estimate the cost of one task based on the cost of another or some technical characteristics
Mapping cost estimate to a schedule model is simplified by: Unifying cost (often product based) and schedule (often task based) work breakdown structures Specifying Time Dependent and Time Independent costs and their uncertainty separately Defining how the TI or TD cost is phased over the task duration
Mapping of Cost to ScheduleMapping of Cost to Schedule
TD Cost
TD Phasing
Total Cost
TI Cost
TI Phasing
28 Aug 2012 PRT-148 Approved for Public Release 13 of 35
The JACS Process:Mapping the Risk Register to Schedule Tasks
The JACS Process:Mapping the Risk Register to Schedule Tasks
Risk
Sched
Cost
CollectSched Data
Create AnalysisSchedule
Collect Risk Data
Assign Likelihood,
EstimateImpact
Map toSched
Activities
UpdateAnalysisSchedule
Collect Cost Data
Identifyas
TD or TI
Map toSched
Activities
28 Aug 2012 PRT-148 Approved for Public Release 14 of 35
Defined as: If risk event A occurs, there is a cost consequence or opportunity. The probability of A occurring is x% Often modeled as a separate task inserted into the schedule network
If there are only a few such risk events, treat as discrete what-if cases (event cost or schedule impact is either “in” or “out” of the estimate) Point estimate includes the full impact (when there are few)
If there are many such risk events, model using the Yes/No distribution (also known as the Bernoulli distribution) When there are many, there is no standard on how to treat the Point Estimate:
Include none and assess separate from the Point Estimate?
Include all (worst case scenario)?
Include the sum of the expected values? this is a common approach
The risk register should account for: Uncertainty of the cost consequence or opportunity Correlation across duration and cost uncertainties Probabilistic branching rarely attempted
Discrete RiskDiscrete Risk
28 Aug 2012 PRT-148 Approved for Public Release 15 of 35
5 0 0 1 2 1
4 0 0 0 1 0
3 1 2 1 4 0
2 1 3 4 2 4
1 0 1 2 2 4
1 2 3 4 5Likelihood
Total Risks = 36
High = 13
Medium = 13
Low = 10
5x5 Matrix Definitions*5x5 Matrix Definitions*
Risk Management conventions* Consequence
1 Minimal or no impact 2 Additional resources < 5% 3 Additional resources = 5-7% 4 Additional resources = 7-10% 5 Additional resources > 10%
Likelihood of Occurrence 1 Remote (10%) 2 Unlikely (30%) 3 Likely (50%) 4 Highly likely (70%) 5 Near certainty (90%)
Opportunities Should have a separate matrix to
address potential opportunities to save (not addressed in our example)
*Note: Taken from Risk Management Guide for U.S. DoD AcquisitionEvery Agency will set its own standards for these values
Co
nse
qu
en
ce
28 Aug 2012 PRT-148 Approved for Public Release 16 of 35
The JACS Process:Assigning Uncertainties
The JACS Process:Assigning Uncertainties
Risk
Sched
Cost
CollectSched Data
Create AnalysisSchedule
Collect Risk Data
Assign Likelihood,
EstimateImpact
Map toSched
Activities
UpdateAnalysisSchedule
Collect Cost Data
Identifyas
TD or TI
Map toSched
Activities
AssessDuration
Uncertainty
AssessCost
Uncertainty
Assess Event Cost and Duration
Uncertainty
28 Aug 2012 PRT-148 Approved for Public Release 17 of 35
Most common method to address uncertainty is to assign distributions to uncertain elements and run a Monte Carlo simulation Objective Uncertainty Distributions
Derived from historical data Something you can defend mathematically and historically
Subjective Uncertainty Distributions Based more on expert opinion than statistical analysis Often necessary due to lack of information to characterize it objectively
Every duration, cost and consequence in the model is generally an estimate and therefore uncertain Time Independent Cost Uncertainty Time Dependent Cost (Burn Rate or Resource Utilization) Uncertainty Duration Uncertainty Discrete Risk Uncertainty –Probability of Occurrence
Uncertainty should be applied in a consistent manner across the entire model
The Only Certainty is UncertaintyThe Only Certainty is Uncertainty
28 Aug 2012 PRT-148 Approved for Public Release 18 of 35
Typical Uncertainty DistributionsTypical Uncertainty Distributions
DISTRIBUTION TYPICAL APPLICATIONKNOWLEDGE OF
MOST LIKELY
NUMBER OF PARAMETERS
REQUIREDRECOMMENDED PARAMETERS
Lognormal
Default when no better info.Probability skewed right.
Replicate another model result.Power OLS CER uncertainty.
Mean or median known better than the
mode (most likely)2
50% (median) and high value
(some tools have a 3rd parameter : “Location” . By default, it is zero. Used to “slide” the
lognormal left or right (even into negative region).
Triangular
Expert opinion. Finite min/max. Chance reduces towards
endpoints. Skew possible.Labor rates, labor rate
adjustments, factor methods
Good idea 3 Low, mode, and high
BetaPertLike triangular, but treats mode as 4 times more important than
min or max.Very good idea 3 Low, mode, and high
BetaLike triangular, but min/max
region known better than mode.Not sure
4 Min, low, high, and max
NormalEqual chance low/high.
Unbounded in either directionLinear OLS CER uncertainty.
Good idea, but unbounded in either
direction2
Mean/Median/Mode and high value
UniformEqual chance over uncertainty
range. Finite min/max.No idea 2 Low and High
(some tools require min and max)
Note: Low/high are defined with an associated percentile (by default 15/85). Min/Max are the absolute lower/upper bound (also known as the 0/100). Some policies require truncation at zero.
28 Aug 2012 PRT-148 Approved for Public Release 19 of 35
The Double Counting Time Dependent (TD) Cost Uncertainty Dilemma
The Double Counting Time Dependent (TD) Cost Uncertainty Dilemma
Consider how TD cost is calculated Typically calculated using: Duration * Cost/Day (duration in days) Cost/Day can be derived from similar, completed project totals Cost/Day factor may already capture cost and duration uncertainty Uncertainty on Duration and Cost/Day factor may be double counting
However, basis for Cost/Day must be carefully understood Cost/Day factor may change as the duration changes
Shorter duration achieved by using more resources ($/day larger) Shorter duration achieved by using more expensive resources ($/day larger) Longer duration a consequence of scarce resources ($/day smaller)
In these contexts, uncertainty on Duration and Cost/Day is appropriate Correlation between the two should be considered as well
28 Aug 2012 PRT-148 Approved for Public Release 20 of 35
The JACS Process:Apply Correlation, Validate then Run
The JACS Process:Apply Correlation, Validate then Run
Risk
Sched
Cost
CollectSched
Data
Create AnalysisSchedule
Collect Risk Data
Assign Likelihood,
EstimateImpact
Map toSched
Activities
UpdateAnalysisSchedule
Collect Cost Data
Identifyas
TD or TI
Map toSched
Activities
AssessDuration
Uncertainty
ValidateFile
RunAnalysis
AssessCost
Uncertainty
Assess Event Cost and Duration
Uncertainty
28 Aug 2012 PRT-148 Approved for Public Release 21 of 35
JACS models developed in any tool will have limited “functional” correlation between uncertain elements (correlation due to model mathematics)
Consider applying correlation to ensure elements that should, do move together
Tools should allow application of correlation across any uncertain elements Just because you can apply correlation, does not mean you should! Correlating Dur with TD may be double counting if TD is modeled as a function of Duration
Ideally, measure the correlation present in the model first, then apply as needed
Apply CorrelationApply Correlation
28 Aug 2012 PRT-148 Approved for Public Release 22 of 35
Prior to running an integrated simulation, the user should review the file to ensure that there are no potential issues within the file
There are many commercial tools that will perform a schedule health assessment These tools look for violations of schedule model best practices (e.g., task without predecessor)
A JACS Health check is more comprehensive and should also uncover issues such as: Critical issues: e.g., cost not phased, invalid uncertainty, duration with no cost, invalid correlation Warnings: e.g., uncertainty on zero cost, risk event with zero probability, baseline outside uncertainty Information: e.g., extraordinary float, risk event turned off, duration without uncertainty, no
correlation
Perform a Comprehensive Health Check to Ensure Model Validity
Perform a Comprehensive Health Check to Ensure Model Validity
28 Aug 2012 PRT-148 Approved for Public Release 23 of 35
The JACS ProcessThe JACS Process
Risk
Sched
Cost
CollectSched
Data
Create AnalysisSchedule
Collect Risk Data
Assign Likelihood,
EstimateImpact
Map toSched
Activities
UpdateAnalysisSchedule
Collect Cost Data
Identifyas
TD or TI
Map toSched
Activities
AssessDuration
Uncertainty
ValidateFile
RunAnalysis
AssessCost
Uncertainty
Assess Event Cost and Duration
Uncertainty
28 Aug 2012 PRT-148 Approved for Public Release 24 of 35
What is JACS?
Overview of the JACS modeling process
Key reports from a well constructed JACS model
Concluding remarks
AgendaAgenda
28 Aug 2012 PRT-148 Approved for Public Release 25 of 35
Typical Output from a Cost Only Uncertainty Analysis
Typical Output from a Cost Only Uncertainty Analysis
Cost uncertainty is generally performed on total costs
Rarely linked to schedule uncertainties Unable to relate a specific cost result to a specific schedule result
No insight into uncertainty by year, or the impact of schedule slips
28 Aug 2012 PRT-148 Approved for Public Release 26 of 35
A JACS Model Relates Uncertain Cost with Uncertain Duration
A JACS Model Relates Uncertain Cost with Uncertain Duration
New View Shows Cost Aligned with Schedule
70% Cost Confidence Level (CCL) Indicates Reserves Capture Schedule Growth
Lower left identifies joint probability of meeting BOTH
70% cost and schedule (58.4%)
Each dot is the cost and
schedule result from
one trial
28 Aug 2012 PRT-148 Approved for Public Release 27 of 35
Annual Funding ChartsAnnual Funding Charts
Point Estimate vs Annual Uncertainty
The JACS Model provides total and annual uncertainty results Identifies both WHAT the uncertainty is and WHEN it is
28 Aug 2012 PRT-148 Approved for Public Release 28 of 35
Additional Views Are PossibleAdditional Views Are Possible
Point Estimate similar to Mean
Need to use Reserves to move point estimate towards the Mean
28 Aug 2012 PRT-148 Approved for Public Release 29 of 35
Additional Views Are PossibleAdditional Views Are Possible
There are many tools available and they all produce some version of most of these charts
28 Aug 2012 PRT-148 Approved for Public Release 30 of 35
The term “Tornado” is used in a variety of contexts across schedule, cost and general uncertainty analysis; tool documentation and the literature
Various definitions include: Find Cost Drivers: Create a low and high “what-if” case for every “cost driver” based on
their 10/90% values (evaluated one at a time)…find driver that has most impact Problem: ignores correlation effects; does not address drivers influenced by multiple
distributions
Find Uncertainty Drivers: Measure the correlation between each defined input distribution and the output of interest. The highest correlation identifies find distribution that has the most impact on the total uncertainty.
Problem: Element that is highly correlated with output may have nothing to do with that output
Hybrid: Sort the trial results by cost driver (one at a time) to find associated bounds on the output mean or selected percentile
Problem: How you bin the trials has massive effect on results; may require huge number of trials to obtain stable results
Brute Force: Use one of the above methods to find the top 10, then run the simulation 10 times turning one at a time and record the impact on the output of interest.
Problem: Very tedious, time consuming and may be misleading if its loss triggers unexpected conditions in the simulation
Conclusion: Beware of the Tornado chart!
Most Misunderstood Chart:Tornado
Most Misunderstood Chart:Tornado
28 Aug 2012 PRT-148 Approved for Public Release 31 of 35
Total Cost Drivers: (left chart) Measure impact on inflated results, not constant year cost results Run simulation to find combined uncertainty results for 10/90%
Total Uncertainty Drivers: (right chart) Adjust the algorithm to account for applied correlations Identify how many trials used (demonstrate sufficient)
Notice the answers are quite different
Tornado Chart Best PracticesTornado Chart Best Practices
7k Trials Required
28 Aug 2012 PRT-148 Approved for Public Release 32 of 35
Prototype Integrated Time-Based View of Costs, Schedule, Budget, and Risks
Prototype Integrated Time-Based View of Costs, Schedule, Budget, and Risks
Solid line is the point estimate (BCWS) Dots are cost/schedule uncertainty results at various milestones X- Axis identifies when key risk register events occur
Size of symbol indicates impact, color indicates probability (or type)
28 Aug 2012 PRT-148 Approved for Public Release 33 of 35
What is JACS?
Overview of the JACS modeling process
Key reports from a well constructed JACS model
Concluding remarks
AgendaAgenda
28 Aug 2012 PRT-148 Approved for Public Release 34 of 35
What Data Do We Need?What Data Do We Need?
Keep it simple and use what you have
Schedule + Risks + Costs = JCL Model Can use any of the below, as long as you have one data source for each
category Schedule
Detailed IMS Simple schedule with just a few moving parts
Costs – preferably time phased Budget data Lower level cost data (LCC databases) / EVM data Parametric costs
“Risks” Risk management system What –if’s Basic uncertainty
28 Aug 2012 PRT-148 Approved for Public Release 35 of 35
Why JACS is Becoming PopularWhy JACS is Becoming Popular
Delivers an integrated view to Project Managers: Schedule probability of success Cost probability of success Impact of discrete program risks Results of any number of what-if scenarios Both total and annual funding reserve requirements
For NASA: Regulatory requirement (7120.5 E) Identify a cost and schedule range by milestone KDP (~milestone) B Baseline program to a specific joint probability level by KDP
(~milestone) C