Greathouse

JCL Status of Implementations-IPAO Programmatic Assessment Philosophy

Mr. Richard Greathouse, PAG Independent Program Assessment Office

Seventh Annual NASA Program Management Challenge 2010

February 10, 2010NASA Internal Use OnlyUsed with permission

Introduction

• Last year’s presentation reviewed how Programs and Projects were to address Joint Confidence Level (JCL) requirements as required by NPD 1000.5.

• This presentation will report on the status of the implementation of the JCL policy. It will cover challenge of integrating the cost and schedule analysis with risk assessments to provide a JCL product.

NASA Internal Use Only

NPD 1000.5

• (a) Programs are to be baselined or rebaselined and budgeted at a confidence level of 70% or the level approved by the decision authority of the responsible Agency-level management council. For a 70% confidence level, this is the point on the joint cost and schedule probability distribution where there is a 70% probability that the project will be completed at or lower than the estimated amount and at or before the projected schedule. The basis for a confidence level less than 70% is to be formally documented.

• (b) Projects are to be baselined or rebaselined and budgeted at a confidence level consistent with the program’s confidence level.


P/p and SRB Roles andResponsibilities

• Both P/p and SRB will develop probabilistic cost and schedule estimates

• Both P/p and SRB will perform cost and schedule risk analysis

• Both P/p and SRB will develop JCL Assessment

• SRB will evaluate the P/p products

• P/p and SRB results will be shown at the governing PMC


Program/project Products

• Probabilistic Cost Estimate

• Integrated Master Schedule Tied To Cost Estimate

• Schedule Risk Analysis

• JCL Assessment


SRB KDP Products

• Benchmark Cost Estimates (for KDP B & C)

• Benchmark P/p Schedule Assessment/Risk Analysis

• Independent JCL Assessment

• Above “yardstick” enables the SRB to evaluate P/p products:

• Cost Estimate• Schedule Risk Analysis• JCL


Programs/Projects

• Completed JCL’s to-date:• LDCM• MMS• NuStar• MSL

• In addition, an Integrated “Cost Given Schedule”risk assessment was performed for GPM


General Feedback

• Insufficient resources for building the JCL Model• Resource-loaded integrated master schedules were not available

(about ½ of total effort)• Cost, Schedule and Risk were not linked at any level (about ¼ of

total effort)

• Cost and Risk not captured at the same level of detail as the schedule

• Excursions took hours to run with MS Project and @Risk

• Work-arounds are in progress

• Weak basis for uncertainty factors used at the subsystem level


General Feedback (Cont.)

• SRB process for evaluating the JCL was not well defined

• Role of benchmark cost and schedule estimates in helping to evaluate the JCL is not clear

• Processes for vetting the JCL technical inputs through the SRB is somewhat confused

• The JCL effort required 3-8 staff months to prepare

JCL is still going thru some rough growing pains.


JCL Implementation Results

•LDCM

• GPM (SRB Integrated Cost/Schedule Analysis)

• MMS


Time and Resources for JCL

• LDCM spent over 1 person-year to complete JCL and reach consensus with SRB

– Split ~50/50 between technical/management and scheduling/JCL personnel– Requires senior project official to lead/coordinate at average of 0.2 rate– Requires the most critical & knowledgeable people on the project for risk analysis– Routine JCLs would require additional scheduling support and ready access to JCL

experts

• Install tool and perform trial run took ~3 months• Actual JCL took ~2 months

– Facilitated by risk analysis from trial run

• Refinement with SRB took ~1 week• Next JCL will probably take ~3 months

– Including one month allowance for:• Full risk analysis• Re-familiarization with process/tools


LDCM JCL Lessons Learned

• Projects must have training and perform trial run JCL before running first real JCL

• Implementation plans and schedules must be very healthy to run JCL– PDR-level plans/schedules may not be sufficient due to open trades and

lack of schedule refinement

• JCL cannot be done without JCL expert support– Projects do not have sufficient statistical analysis expertise

• LDCM JCL results driven mostly by uncertainty factors, not risks– 10 of the 12 months of LRD movement and 80% of cost growth attributed to

uncertainty factors• Probably due to relatively low risk likelihoods because we mitigate

known risks

• Basis of estimate for uncertainty factors is currently not well developed– Active area of research in cost estimating community– Do uncertainty parameters double-book the impact of identified risks?

• After 200 iterations, 70 percentile solution stabilized to approximately +/- $15M and +/- 3 weeks


LDCM JCL Lessons Learned (cont.)

• Risk analysis – Requires significantly more work and speculation than required for effective continuous

risk management– Value of detailed risk analysis in question since risks had little impact on LRD or cost

• IMS model for JCL cannot be maintained, it must be recreated every time a JCL is needed– New schedules delivered monthly by contractors– Need to remove reserves, prune unnecessary detail, assess and link in risks, cost load,

etc.

• JCL is not a useful tool for assessing exposure to a specific risk or trade mitigation options– Impact of any particular risk is dwarfed by uncertainty in risk assessment and impact of

general uncertainty factors

• JCL cannot model IMS logic changes and work-arounds that would actually occur when risks are realized

– Must be estimated when risk impacts are assessed

• JCL tool used by LDCM has significant weaknesses– Required significant degree of custom coding by JCL experts to implement JCL– Macros and settings that run JCL are not saved uniquely for each run file


Other Observations

• Yin– JCL process forces project to analyze schedule, risks, and costs to a much higher degree than would otherwise

be done– Unlike parametric ICE, JCL better captures unique aspects of a project– JCL under project control to ensure accuracy of inputs/assumptions, and is objectively evaluated by SRB– 70% values probably are conservative cost and schedule estimates

• But are only “neighborhood” estimates, not a prediction– JCL results can be credibly “justified”

• Yang– JCL requires high degree of speculation on risks, impacts, workarounds, and uncertainties

• The risks that bite you are the ones you don’t expect, and are therefore only covered by nebulous uncertainty factors

– JCL cannot anticipate disaster scenarios and unforeseen political impacts to cost and schedule• No cost/schedule analysis tool can• JCL cost/schedule estimates are no more accurate than any other educated guess

– JCL process offers an infinite number of nebulous knobs to turn• Both the project and the SRB have significant latitude to justifiably tailor results with high degree of

impunity

Bottom Line: Despite its many shortcomings, a JCL developed by the project and reviewed by the SRB is probably the best known way to methodically establish a reasonably-conservative justifiable

cost/schedule estimate that has the support of both the project and independent reviewers


LDCM JCL Observations

• It is important to have an IMS first. The IMS needs to be reviewed and pass all health checks before any risk or uncertainties are added. If the schedule is not any good, then the JCL results are worthless.

• Uncertainties are driving the JCL, not the risks themselves. The uncertainties are based on DoDclassified data, and may or may not be relevant to NASA spacecraft. Some knowledgeable SRB members feel that the DoD experience is not equivalent to the LDCM spacecraft.

• The IMS is not a traditional “resource loaded schedule”. The resources applied are not labor and material costs by activity, but an allocation of the budget dollars. Reasonable assumptions have been made to derive the fixed costs (mostly material costs and level of effort activities) and variable costs (labor) for the budget. Only TIRS and OLI allocate the budget to the next level of the WBS (e.g., FPA, Structure, etc.) The remaining budget is allocated at the higher WBS Level (e.g. spacecraft, launch vehicle, EPO, etc.)

• The budget is a “given” and assumed to be a good “cost estimate.” Therefore, any 70th percentile JCL is grounded in acceptance of the budget.

• JCL must be maintained. This requires effort, skilled people and senior project official to lead and coordinate JCL.

• Most Project Offices will have to purchase additional tools and training, and hire expert support for statistical analysis.

• JCL process does force Projects to analyze schedule, risks, and costs to a much higher degree than would otherwise be done. [Jenstrom, LDCM JCL Observations]

• LDCM Project Office bottom line was as follows: “Despite its many shortcomings, a JCL developed by the project and reviewed by the SRB is probably the best known way to methodically establish a reasonably-conservative justifiable cost/schedule estimate that has the support of both the project and independent reviewers.” [Jenstrom, LDCM JCL Observations]



• LDCM

•GPM (SRB Integrated Cost/Schedule Analysis)• MMS


• Cost S-curve becomes steeper with increased certainty of the project’s duration.

• Two plots of P (Cost < x | Schedule < y) the symbol, | , means “given”

• The blue curve is the original Cost S-curve, i.e., P (Cost < x | Schedule < )

• The pink curve is the modified Cost S-curve, given that we know that the launch will occur before 01 Oct 13.

∞

P

TY$M

GPM Conditional Cost Distribution


GPM Observations

• Overall cost and schedule risk is reduced by observing the interaction between cost and schedule.– Conditional Probability of Cost given Schedule

– Joint Cost and Schedule Probability

• Correlation between cost and schedule can be modeled in different ways:– Parametric model vs. Resource-loaded schedule model

GPM Observations

• Parametric Cost/Schedule Integration Model– Calculations of Conditional S-curves done using the NASA Cost-

Schedule Integration Spreadsheet (MCR, Inc.)

• Requires five parameters:

– Mean and standard deviation from traditional Cost S-curve

– Mean and standard deviation from traditional Schedule S-curve

– Cost and Schedule correlation coefficient of 0.8 based on analysis by Aerospace Corp.


• LDCM

• GPM (SRB Integrated Cost/Schedule Analysis)

•MMS


MMS JCL Lessons Learned (1)

• The MMS Project JCL model implementation was too complex in its construction, inefficient in its execution and will not satisfy future Project needs for rapid, insightful “what if” analysis and reporting.

• The PRSL model developed by the SRB using MS-Project with @Risk with an Excel JCL add-on performed well.– Simple model construction ( 165 total tasks for cost,

schedule and discrete risks).– Validated the Project baseline (matched current plan with

reserves removed)– Documented basis for uncertainty ranges selected and

discrete risk branching (likelihood, cost & schedule impacts)– Rapid execution (complete JCL “what if” update cycle of 40

minutes)


• The results of the JCL are highly dependent on the assumptions made about the risk distributions to be used and what types of discrete risks should be included in the analysis. In order for future JCLs to provide “apple to apple” comparisons across projects, process standardization (normalization) should be provided by PA&E in the form of a published guideline.

– MMS Project used narrow, symmetric triangular distributions for modeling risk and uncertainty. SRB used broader, right skewed triangular distributions that are more representative of aerospace industry norms.

– MMS Project included discrete risks that were already covered by uncertainty (i.e., double accounting of risk). This did compensate somewhat for their narrower probability distributions.

– MMS Project included discrete risks that were external in origin and outside their span of control, though likely to have a negative impact on performance. The SRB chair regarded these risks as “out of scope” for SRB deliberations. A policy decision is needed on the handling of external project risks in the JCL.


MMS JCL Lessons Learned (1)

Overall Recommendations

– Continue development of good cost and schedule tools.

• Schedules to-date often do not pass basic health checks.

– Use native data formats.

– Develop a smooth transition to a workable integrated methodology.


JCL is a good product, yet to come of age.

Greathouse

Technology

Transcript of Greathouse