Life Cycle Review - Programmatic Analysis Lessons Learned

NASA PM Challenge – February 2012

Richard Greathouse, NASA IPAO

Heidemarie Borchardt, PMP, NASA IPAO

Michele King, Reed Integration

Kelly Moses, PE, Reed Integration

Robin Smith, Reed Integration

Overview

• Programmatic analysis lessons learned in preparing for a Life-Cycle Review (LCR) will be presented. Standing Review Board (SRB) Timeline Data Deliveries

• Topics that will be covered include the SRB independent analysis process and methodology: Basis of Estimates (BOE) Schedule Analysis Reserve Analysis Range Estimates and Joint Confidence Levels (JCL) Resource Analysis Uncertainty and Risk

 

2

Programmatic Analysis

3

Criteria Rating Rationale

Adequacy of management approach.

Alignment with and contribution to Agency needs, goals, and objectives, including the adequacy of requirements flow-down.

Adequacy of technical approach, as defined by NPR 7123.1 entrance and success criteria.

Adequacy of the integrated cost and schedule estimate and funding strategy in accordance with NPR 1000/5.

Adequacy/availability of resources other than budget.

Adequacy of risk management approach and risk identification and mitigation per NPR 8000.4.

4

Data Deliveries

5

• Documents (Requested of Projects) focusing on: Project Management Plan, Risk Management Plan, Schedule Management Plan Work Breakdown Structure (WBS) and WBS Dictionary Resources Plan (staffing, workforce, facilities) Project management status reports and EVM Data

• Master Project Schedules and P/p Integrated Master Schedule (IMS) Program runs STAT health check and schedule should be green for delivery Identification of Critical Path and Schedule Margin/Reserve with basis Removal of constraints from schedule for purposes of performing SRA Provide basis for any constraints not removed from schedule Identification of risk mitigation activities in the schedule Qualitative uncertainty ratings by WBS (Ref. schedule risk consequence chart)

• Risk List Likelihood and Consequence Probabilities for Risks Mapping of Risks to Specific Schedule Activities

Alignment with and Contributing to Agency Needs, Goals, and

Objectives, Including the Adequacy of Requirements

Flow-Down

6

Needs, Goals, Objectives and Requirements Flow-Down

7

• Mission success criteria identified.• Clear and consistent requirements.• Requirements flow-down.• Dependencies on other program requirements

within NASA or outside.

Adequacy of Management Approach

8

Management Approach

9

• An assessment of management processes including acquisition strategy, budgeting, tracking, reporting and control. Analysis will provide insight into trends, problems issues and risks so that mitigation strategies can be developed.

• Analysis includes an assessment of the acquisition plans, including major contracts, scope, and contract types.

• Cost and schedule baseline and Earned Value Management (EVM) analysis.

• Reporting process (per Program Plan): monthly status reports.

• Adequacy of staffing, training, communications, information technology and resources.

• Review of management processes: Program Plan, Program Commitment Agreement, Work Breakdown Structure (WBS) and WBS Dictionary, Schedule Management Plan.

Management Approach Lessons Learned

10

• Important to meet with project’s programmatic team as part of the SRB review process Risk Manager Scheduler Business Manager

• Open communication facilitates the review process. SRB experts can provide valuable feedback to P/p prior to LCR.

Alignment of Technical Approach, as Defined by NPR 7123.1

Entrance and Success Criteria

11

Technical Approach

12

• What are the technical drivers for cost, schedule, and risk?

• Are technical margins adequate?• What is the Technology Readiness Level (TRL)

and are there issues in meeting the TRL?• What are the heritage assumptions, rework, and

sparing plans which can affect design, manufacturing, and test cycles?

• Are test plans / testing adequate?

Technical Approach –Lessons Learned

13

• Allowing the SRB to participate and have insight into the full set of instrument and subsystem reviews leading to the Site Review: Minimizes the prospect of a steep learning curve at the Site

Review. Allows time to discuss any findings among the SRB prior to the

Site review. Provides the SRB with insight into the issues across the project

and aids in identification of systemic issues.

Adequacy of the Integrated Cost and Schedule Estimate and

Funding Strategy in Accordance with NPD 1000.5

14

Analysis & Methodology

15

• SRB will assess the project’s cost and schedule estimate.• SRB will evaluate risks and derive impacts against cost

and schedule.• The SRB will employ its expertise to evaluate

Program/project (P/p) risks and will require the P/p cost/schedule models to respond to their inputs for risk and uncertainty.

• SRB will evaluate the reserve and lien posture of the P/p.• SRB will use the Independent Programmatic Assessment

(IPA) to derive impacts of uncertainties and risks to cost and schedule.

Analysis & Methodology - Lessons Learned

• Early, frequent, and active communication and dialogue between the SRB and the project, consistent with the SRB timeline, facilitates the understanding of the project’s analysis and methodology regarding its integrated cost and schedule estimate.

16

Basis of Estimate

• Where does the project get this information?• Does it explain “why” the estimate is what it is? (Is a basis

provided? Is it traceable?)• Is the explanation credible? (Is it based on expert opinion,

heritage, calculations, etc?)• Is it realistic? Is it based on the complete scope of work?• If it’s cost, is it time-phased?

17

Basis of Estimate – CostLessons Learned

18

• The BOE should document the estimating methodology used to prepare the cost and schedule estimates for each WBS element.  The content of the BOE depends on the estimating method used.  The four most common methods are: Time and Material Estimates – the BOE should describe in detail how the effort

and the duration were determined. Parametric Estimates – the BOE should describe any cost or schedule models

used and how they were applied.  Analogy Estimates – The BOE should describe the analogous system(s) and

explain how and why it was used, explain any scaling or adjustments made to the data for the analogy. The BOE should document any historic schedule plan/actual data from analogous missions or projects that were used in the estimate.

Engineering Judgment – The BOE should describe the thought process and justification for an Engineering Judgment type of estimate.

Basis of Estimate Evaluation

19

DefinedWBS WBS WBS Formal Scope/ Schedule Basis Time

Number Level Element BOE TraceableBaseline Durations Estimating Methodology Provided Complete Accurate Realistic Phasing Risk Methodology123456 1 Project - - - - - - - - - - - - - - - - - - - - Summation - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -123456 2 Project Management - - - - - - - - - - - - - - - - - - - - Summation - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -123456.01.013 Project Management G G Y R Time & Materials G G Y R G R123456.01.023 Business Management Time & Materials123456.01.033 Risk Management Time & Materials123456.01.043 Procurement Management Time & Materials123456.01.053 Facilities Management Time & Materials123456 2 Systems Engineering - - - - - - - - - - - - - - - - - - - - Summation - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -123456.02.013 Systems Engineering Management Time & Materials123456.02.023 System Requirements Time & Materials123456.02.033 System Interface & Configuration Engineering Build-Up123456.02.053 System Verification and Validation Time & Materials123456.02.063 Trade Studies Time & Materials123456.02.073 Systems Risk Management Plan Time & Materials123456 2 Safety and Mission Assurance - - - - - - - - - - - - - - - - - - - - Summation - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -123456.03.013 Safety & Mission Assurance Management Time & Materials123456.03.023 System Safety Time & Materials123456.03.033 System Reliability Time & Materials123456.03.043 Quality Assurance Time & Materials123456.03.053 Environmental Safety Time & Materials123456 2 Science and Technology - - - - - - - - - - - - - - - - - - - - Summation - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -123456.04.013 Science and Technology Management Analogy123456.04.023 Science Requirements Analogy123456.04.033 Science Development Analogy123456 2 Payloads - - - - - - - - - - - - - - - - - - - - Summation - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -123456.05.013 Payloads Management Time & Materials123456.05.023 Payloads Requirements Time & Materials123456.05.033 Payload #1 Development - - - - - - - - - - - - - - - - - - - - Summation - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -123456.05.03.014 Payload #1 Subsystem #1 Development Analogy123456.05.03.024 Payload #1 Subsystem #2 Development Analogy123456.05.043 Payload #2 Development - - - - - - - - - - - - - - - - - - - - Summation - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -123456.05.04.014 Payload #2 Subsystem #1 Development Parametric CER123456.05.04.024 Payload #2 Subsystem #2 Development - - - - - - - - - - - - - - - - - - - - Summation - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -123456.05.04.02.015 Payload #2 Subsystem #2 Component #1 Development Parametric CER123456.05.04.02.025 Payload #2 Subsystem #2 Component #2 Development - - - - - - - - - - - - - - - - - - - - Summation - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -123456.05.04.02.02.016 Payload #2 Subsystem #2 Component #2 Part #1 Development Engineering Build-Up123456.05.04.02.02.026 Payload #2 Subsystem #2 Component #2 Part #2 Development Engineering Build-Up123456.05.053 Payload Systems Integration & Testing Analogy123456 2 Spacecraft - - - - - - - - - - - - - - - - - - - - Summation - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -123456.06.013 Spacecraft Management Factor123456.06.023 Propulsion Analogy123456.06.033 Architecture Parametric CER123456.06.043 Avionics Parametric CER123456.06.053 Thermal Control Parametric CER123456.06.063 Software Parametric CER123456 2 Mission Operations - - - - - - - - - - - - - - - - - - - - Summation - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Basis of Estimate – Schedule Lessons Learned (1)

20

• If 3-point estimates are provided, then the BOE should include the rationale for estimating the 3 points.

• The BOE should discuss whether or not the task duration is a function of the level of effort or resources applied to the task, e.g. if working overtime or multiple shifts will accelerate the task.

• Activity owner should make the duration estimate for the task.

• Durations should be estimated for most likely time values and should be representative of timecard accounting periods or less than 2 months. If durations are not realistic, deliverables and critical path will be incorrect.

• LOE or hammock tasks should be clearly identified and justified. 

Basis of Estimate – Schedule Lessons Learned (2)

21

• Estimates should not be padded and should reflect required effort to complete the activity, resources available.

• Activity durations and cost estimates should map to the corresponding WBS; FTEs/WYEs requirements support activity and cost estimate.

• Schedule estimates may be based upon historical data, parametric analysis, and subject matter experts.

• Document the basis of the estimate: “Documenting the basis for the duration facilitates communication of expectations between activity owners and senior decision makers, and aids in the estimating of future analogous activity durations.” – GAO

Basis of Estimate – Schedule Lessons Learned (3)

22

• A good BOE should not only document the basis for the activity duration, but also:Mandatory logicConstrained datesLeads and LagsCalendarsResources

• Scheduler documents BOE rationale in MS Project text field.

• Durations of other missions can be used as part of the BOE.

Estimating - Live Demo

23

Note – these people are not actors; they are professionals!

Schedule Analysis - Lessons Learned

24

• Schedules may pass the health checks but still have flaws; focusing solely on schedule mechanics will not fix a poor quality schedule.

• Schedules that fail the health checks should be corrected prior to performing integrated cost and schedule analysis (Note: If the true critical path cannot be identified in the schedule before the range estimate or JCL activity, then the resulting critical path will not be credible).

• Analysis schedules should be developed to a level that will incorporate any identified risks that the project is carrying.

• Schedule trends for milestones, deliverables, and interfaces need to be analyzed and monitored.

Reserve Analysis – Lessons Learned

25

• Different NASA centers and organizations apply and manage reserve differently.

• It is important to understand the basis for the reserves and how the P/p is managing reserve.

• Margin or reserve tasks should also be clearly identified and the method used to estimate margin should be explained.

• References: Schedule Management Handbook JPL Rules GSFC Gold Rules

Range Estimate / Joint Confidence Level (JCL)

Lessons Learned

26

• Importance of analysis schedule reflecting true critical path, logical linkages and enough detail to map risks and cost elements.

• Importance of working with cost, schedule and risk analysts.

• Tools should not affect the process.• Understanding and communicating results.

Adequacy/Availability of Resources other than Budget

27

Resources

28

• Analysis includes an assessment of workforce planning, skill mix, retention, obsolescence, and ramping issues.

• Assessment of availability of facilities, availability, costs and supporting infrastructure.

• Analysis of any mission partner’s potential impact to cost and schedule.

Resources - Lessons Learned

• A ramp-up or ramp-down of workforce needs to be consistent with the work flow (i.e. the project schedule/project plan).

• Projects often rely on the same facilities for testing. Schedule slips in one project can cause delays for other projects. In many cases, problems may be avoided if work-arounds are identified early.

29

Adequacy of Risk Management Approach and Risk Identification

and Mitigation per NPR 8000.4

30

Risk Management Approach

31

• Assessment of BOEs for any cost or schedule mitigation plans for the identified risks.

• SRB supported assessment of the risks which may include new risks that are identified by the SRB with associated cost and schedule impacts.

• SRB supported assessment of the risks which may included adjustments to assumptions made by the P/p with associated cost and scheduled impacts (TRL and heritage).

• Assessment of the reserve strategy and the current state of the reserves in relation to where P/p are in life cycle.

• Assessment of whether risk has been accounted for in P/p cost, schedule or JCL process.

Risk Management Approach – Lessons Learned

• The Risk Management Plan should accurately reflect the process that is in place, including: How the P/p/subsystem levels coordinate and interact. Definitions of the 5x5 scoring criteria. Processes for identifying, tracking, mitigating, controlling, and

documenting risks.

• Risks need to be mapped to the appropriate schedule task(s).

32

Uncertainty

• Planned schedule activity durations are estimations; completion dates appear as definite dates with associated definitive costs. Schedule finish dates and costs are not facts, but projections with

associated uncertainty. Generic risks captured in the project’s risk list, such as “Instrument

Schedule,” cannot typically be mapped to specific work activities in the schedule; these types of risks are actually “uncertainty” and should be captured as such when performing a range estimate or JCL.

33

Uncertainty Lessons Learned

• Uncertainty values are applied to WBS levels within a P/p

• In many cases, project trends (performance over time) can be a good indication of the uncertainty associated with a particular element in a project, especially if discrete risks cannot be identified.

• Understanding the P/p’s uncertainty gives the SRB members a basis for assessing uncertainty durations.

34

Schedule - Uncertainty Distributions

Risk Level Low 10% Mode 50% High 90%

Low 0.984 1.000 1.070

Low+ 0.954 1.000 1.240

Medium 0.924 1.000 1.390Medium+ 0.893 1.000 1.550

High 0.863 1.000 1.710

High+ 0.832 1.000 2.020

Very High 0.802 1.000 2.350Very High+ 0.771 1.000 3.370

Extra High 0.741 1.000 4.384

Risk

• Known risks, identified by Program, Project or SRB are normally called “discrete risks” with corresponding risk mitigation activities. Mitigation may require funds and schedule and the SRB needs to

understand the impact on schedule/cost reserves. Discrete risks should mapped to the schedule activities they affect

and be assigned associated probability and impact values for cost

and schedule.

35

Risk - Lessons Learned

• Projects should be able to quantitatively evaluate their risks, by identifying the following (reference SRB risk chart): Probability of Risk Occurrence Best Case Schedule (Work Days) Most Likely Schedule (Work Days) Worst Case Schedule (Work Days) Best Case Cost ($) Most Likely Cost ($) Worst Case Schedule ($)

• *Note: A basis of estimate should be provided with each of the above quantities.

36

SRB Risk Worksheet

37

Risk Type Identified by Project

(Cost, Schedule,

or Technical)

Rank Risk ID Title Statement L % likelihood rationale Cmin

(optimisitic)min rationale

ml (most likely)

ml rationalemax

(pessimistic)max rationale

Risk Owner

Associated WBS #

Specific Activities Impacted

1

2

3

45

6

7

8

910

11

1213

Risk Type Identified

by SRB(Cost,

Schedule, or

Technical)

SRB-Risk ID

Title Statement L % likelihood rationale Cmin

(optimisitic)min rationale

ml (most likely)

ml rationalemax

(pessimistic)max rationale

Risk Owner

Associated WBS #

Specific Activities Impacted

SRB-1

SRB-2

SRB-3

SRB-4

SRB-5

SRB-6

SRB-7

SRB-8

Consequence (Duration Beyond Already Scheduled [days])Likelihood (% chance of occurring)

Understanding the project’s risk assessment gives the SRB members a basis for their assessment.

Parting Words

• It takes one woman nine months to have a baby. It cannot be done in one month by impregnating nine women (although it is more fun trying). * Spanish Proverb

• The same work under the same conditions will be estimated differently by ten different estimators or by one estimator at ten different times.

• Any project can be estimated accurately (once it's completed).

• A change freeze is like the abominable snowman: it is a myth and would anyway melt when heat is applied.

• Right answers to wrong questions are just as wrong as wrong answers to right questions.

• What you don't know hurts you.• There's never enough time to do it right first time but

there's always enough time to go back and do it again.

• Estimators do it in groups - bottom up and top down. • Good estimators aren't modest: if it's huge they say

so. MHR• If project content is allowed to change freely the rate

of change will exceed the rate of progress.

Reference: http://www.project-training-k.freeserve.co.uk/

• The sooner you begin coding the later you finish.• Anything that can be changed will be changed until

there is no time left to change anything. • Change is inevitable – except from vending

machines. Robert C. Gallagher• The person who says it will take the longest and cost

the most is the only one with a clue how to do the job.

• If you're 6 months late on a milestone due next week but nevertheless really believe you can make it, you're a project manager.

• If you don’t know where you are going any road will take you there. Lewis Carroll (Alice in Wonderland)

• If you fail to plan you are planning to fail.• If you don't attack the risks, the risks will attack you.• A little risk management saves a lot of fan cleaning. • The sooner you get behind schedule, the more time

you have to make it up.• A badly planned project will take three times longer

than expected - a well planned project only twice as long as expected.

38