K gallagher tosborne
-
Upload
nasapmc -
Category
Technology
-
view
13.608 -
download
0
description
Transcript of K gallagher tosborne
Overview of GAO Quicklook
PM Challenge
February 2012
Katie Gallagher and Tracy OsborneOCFO/SID
Outline• Background
– Why is this important? • GAO’s Approach
– What projects are involved?– Why does GAO request technical data?
• What should I expect if my project is in Quicklook?
• What does NASA’s data say?• Agency Messages
1/22/2012 2
Background• The GAO’s Assessment of Large-Scale [NASA] Projects is
Congressionally-mandated
• GAO sends its findings to Congress in an annual “QuickLook Book” (QLB) Report focusing on changes in project cost and schedule and providing GAO’s explanations for these changes
• GAO is frequently called upon to testify about the results.
• Engagement objective: “Identify and gauge the progress and potential risks associated with selected NASA acquisitions.” (i.e., projects)
• GAO uses data from QuickLook Book (QLB) in other audits, such as the ongoing High Risk audit, and vice versa.
31/22/2012
4
Why the Stakeholder Interest?
• NASA’s historical inability to control cost and schedule growth– JWST, CxP, MSL growth cited in several provisions establishing new
reports.– A 2004 GAO finding that “NASA lacks discipline in cost estimation.”– Multiple Operating Plans per year to address cost growth or phasing.
• Based on past Performance NASA is required to report to OMB, Congress and GAO routinely on Project cost and schedule. – All appropriations since FY 2008 have included direction for GAO to
“Identify and gauge the progress and potential risks associated with selected NASA acquisitions.”
1/22/2012
Typical Results from Quicklook Reports
• Most of NASA’s cost growth for the current portfolio is attributed to projects baselined prior to 2009
• NASA projects do not meet GAO-identified best practice standards for technology maturity and design stability
• Other challenges that affect NASA project outcomes include Funding, Launch Vehicle, Development Partners, Parts Quality, Contractor Management
1/22/2012 5
GAO’S APPROACHWhy do they ask about TRL’s and drawings?
1/22/2012 6
GAO’s Report is in Two Parts• Agency Observations
– Examines adherence to NASA policies such as 7120.5E and 1000.5A
– Examines potential systemic reasons for NASA’s cost growth
– Assesses NASA’s efforts to improve cost and schedule growth, including JCL policy
• Project Assessments– Provides an update on status of each project
included in the audit– Outlines challenges specific to each project that
GAO believes may lead to cost growth
1/22/2012 7
GAO Quicklook assesses performance of NASA’s major projects and the Agency’s management of those projects.
How does a project get picked?• GAO evaluates all spaceflight projects in formulation if a contract is
released that has development content and is greater than $25 M– Can include projects in Phase A and Pre-Phase A if they have such a contract
• GAO evaluates all spaceflight projects in development with a life cycle cost greater than $250 M
• GAO continues to track projects in Operations until NASA negotiates removal from Quicklook– Minimal data collection on Operations projects; main focus is cost and
scientific results
• For projects in formulation, GAO expects NASA to provide cost and schedule ranges
• For projects in development, GAO expects NASA to provide baseline cost, schedule, design and technical information
1/22/2012 8
Projects Included in Audit2008 2009 2010 2011
* Estimated Lifecycle Cost >$250M* In development* In formulation if it also has released a development contract valued at >$50M
(consistent with other external reporting requirements)
* Estimated Lifecycle Cost >$250M* In development* In formulation if it also has released a development contract valued at >$25M* If in operations, it is not included
* Estimated Lifecycle Cost close to $250M (if it is called out in the budget as a discreet project then it may be included)* In development* In formulation if it also has released a development contract valued at >$50M * In operations, in the prime mission phase
* Estimated Lifecycle Cost close to $250M (if it is called out in the budget as a discreet project then it may be included)* In development* In formulation if it also has released a development contract valued at >$50M * In operations, in the prime mission phase
1 Aquarius Development X X X X2 Ares Formulation X X X3 Dawn Operations X X X4 Fermi/ GLAST Operations X X X5 GPM Development X X X X6 Glory Development X X X X7 GRAIL Development X X X8 Herschel Operations X X X X9 ICESat II Formulation X X
10 JWST Development X X X X11 J uno Development X X X12 Kepler Operations X X X X13 LADEE Development X X14 LDCM Development X X X X15 LRO Operations X X X X16 MAVEN Development X X17 Mars 2016 (TGO) Formulation X18 MMS Development X X X19 MPCV Pre- Formulation X20 MSL Development X X X X21 NPP Development X X X X22 OCO/ OCO-2 Development X X X23 Orion Formulation X X X24 Phoenix Closed Out X25 RBSP Development X X X26 SDO Operations X X X X27 SLS Pre- Formulation X28 SMAP Formulation X X29 SOFIA Development X X X X30 Solar Probe Plus Formulation X X31 TDRS Development X X32 Wise Operations X X X X
Requirement for Inclusion
Project
Lifecycle Phase as of
Entrance Conference (April 2011)
1/22/2012 9
10
GAO “Business Case” Model
• Knowledge Point 1: Resources and Requirements Match– Indicated by a high level of technology maturity
• Heritage is technology is appropriate in form, fit and function• New Technologies at least at TRL-6 (for NASA)
• Knowledge Point 2: Product Design is stable “about midway through development”– Best practice is to achieve design stability by midpoint of development– GAO cites 90% of engineering drawings releasable as figure of merit
• Knowledge Point 3: Manufacturing Processes are mature– Product shown to be producible within cost, schedule, and quality targets and have demonstrated
reliability and Design demonstrates needed performance through realistic system-level testing
Pre-phase A
Phase A
Phase B Phase C Phase D Phase E
KDP-A KDP-B KDP-C KDP-D KDP-D
PDR CDR SIR
Concept and Technology Development Production
Technology Development Engineering Manufacturing and Development Production and Deployment Operations and
Support
MS-B MS-CPDR
NASA
DOD
GAO
Knowledge Point 1
Knowledge Point 2
Knowledge Point 3
1/22/2012
CDR
Engineering Manufacturing and Development
Production Rarely Applies to NASA Projects
Data Requested AnnuallyData Category Element 2008 2009 2010 2011 Frequency
Design Stability X X X X Annual + UpdatesCritcal Technologies X X X X Annual + Updates
Heritage Technologies X X X Annual + UpdatesSoftware Complexity X X AnnualQuality Parts Issues X X Annual
MPAR Baseline X X X X As OccursARRA Funding X X Semi-annual
KDP-B Est. LCC Range X X X X Semi-annualKDP-C Baseline X X As Occurs
ICEs X X As OccursJ CLs X X As Occurs
Project-held UFE X X Monthly (in MSRs)MD-held UFE by year X Annual
Key Milestones X X X X Semi-annualAgy vs Mgmt Schedule Comparison X Semi-annual
Basic Information X X X X Semi-annualAward Fee Strucuture X X X Semi-annualEVM (CPR Formats) X X Monthly
FAD/PCA X X X X As OccursProject Plan X X X X As Occurs
Control Plans X X X X As OccursPDR/CDR Packages X X X X As Occurs
CADRes X X As OccursSRB Reports X X As Occurs
KDP-C Decision Memos X As OccursKDP-C Datasheets X As OccursMSR Presentations X X Monthly
Documentation
Technical
Cost
Contracts
Reporting Year
Schedule
11GAO reviewed this list for 2011, and agreed to reduce frequency of some requests1/22/2012
WHAT SHOULD I EXPECT?So, my project is in Quicklook…
1/22/2012 12
DCI-C&s
The Quicklook Annual Cycle (2011)
MarchFeb
Entrance Conference
May DecAug
JuneOct
Exit Conference
DCI-proj
AprilJuly Sept
NovJan
Site Visits – JPL &GSFC
Action Item Response
Other Data Requests
2-Pgr Cmts Agency Cmts
Addt’l Data
Addt’l Actions
Final Report
Site Visits – MSFC
&JSCDCI-C&s
DCI-contract
DCI-C&s
DCI = “Data Collection Instrument” Three types:1.Project DCI – for technical data2.Cost and Schedule (C&S) DCI – provided by OCFO3.Contracts DCI – provided by OP
131/22/2012
14
GAO transmits request
• The TPOC and relevant Audit Liaisons (ALR) should be copied on all requests
• PM please forward requests to ALR and QLB TPOC when necessary
Project drafts response
• Provide document and/or explanation
• When document or data is not available, explain why
• Draft response provided to PE, ALR, and QLB TPOC
Response reviewed by HQ & Ctr Mgmt
• Possible reviewers include PE, MD, PCFO, Procurement, OCE, OSMA, LSP
• QLB TPOC and MD ALR will coordinate the appropriate reviews.
QLB TPOC transmits response to GAO
• ALR, PM, PE, and reviewers will be copied
Process for Working with GAO
Standard 3 week turn-around when not otherwise stipulated by GAO
Quicklook Technical POC (TPOC) discusses request with GAO to
understand purpose for requested items
Extra explanation may be provided to help contextualize the data or to better assist the GAO in answering
their questions
1/22/2012
15
• Legislatively, the GAO has right of access to the data, regardless of procurement sensitivity or classification
• NASA can negotiate with GAO regarding the provision of data; if there is a disagreement, GAO’s recourse is legislative action
• GAO is bound by the same rules as NASA for handling the data, and NASA can specify data restrictions
• NASA actively works with GAO on data formats, types and context for interpretation
GAO Data Access Rights
1/22/2012
NASA provides the data requested and works with the GAO to add context and help the GAO analysts to understand NASA’s data.
Data Collection Instruments• GAO asks PMs to fill in a
Data Collection Instrument (DCI) annually and following PDR and CDR
• Data collection includes– Critical and heritage
technology TRLs– Drawing release– Parts Quality issues,
including impact ($)– Software TLOC and
Blocks– Partners
• GAO requests data as of PDR, CDR, and current
• Procurement and OCFO fill in similar DCIs on contracts, cost, and schedule1/22/2012 16
PMs should NOT leave any sections blank; if the data is not available or not applicable, say so and explain why.
Example Technical DCIGAO Data Collection Instrument:
Date Submitted to NASA Project Office:
NASA Project Office Response Required by:GAO Contact:Project:NASA Project Office Instructions
KP ESTIMATE PDR CDR September 2010 update LatestDate 7/12/2010 Scheduled for July 2011 9/16/2010 5/3/2011No. of Releasable
DrawingsNo. of Total
Drawings ExpectedSource Project Office Project Office Project Office Project Office
Key Event Dates Defined: The PDR column should reflect information as of the Preliminary Design Review prior to KDP C (NAR). The CDR column should reflect information as of the Critical Design Review. The "Latest" column should reflect the the number of drawings releasable and expected as of the date this spreadsheet is returned to GAO.
Overall Design Knowledge
See below for Spacecraft and Instruments comments. The project assesses design stability primarily thru the Systems Engineering organization. A weekly Systems Design Team regularly addresses design, design margins, analyses, interfaces, subsystem and instrument design topics, and proposed changes. Proposed design changes are then processed thru standard Configuration Control Boards. In addition, weekly status reviews as well as a Monthly Management Review take place that addresses overall status and issues.
0
1537 sheets expected for spacecraft, 818 drawings expected for instruments.
10 released for spacecraft, 16 released for instruments.
947 spacecraft sheets released to date. 299 drawings released for instruments.
Comments & Issues: Enter comments and issues below or on a separate sheet.
1879 sheets expected for spacecraft.
2437 sheets expected for spacecraft, 1023 drawings expected for instruments.
If the total number of drawings is not known or if the total number of drawings has changed, please provide an explanation in the space below. If the project has completed CDR, please describe any measures and metrics used in addition to engineering drawings to assess design stability. If the project has yet to complete its CDR, please identify the measures and metrics the project intends to use to assess design stability.
2437 sheets of spacecraft drawings planned, 1023 drawings expected for instruments.
Joe AnalystAwesome
Please provide or update the total number of releasable engineering drawings and the total number expected at key dates listed below. The term "releasable" refers to those drawings (physical hard copy or computer-generated packages) that are sent to be manufactured. Please calculate the number of drawings by using the number sheets within a package, not just the number of packages. Sometimes these drawings are considered the "build to" drawings. These drawings include details on the parts and work instructions needed to make the product and reflect the results of testing. Change any dates or data as needed.
Design Stability
5/3/2011
5/6/2011
Site Visits• GAO will prepare for site visits with the projects by reviewing project monthly
status review (MSR or PSR) packages; discussion questions will be transmitted in advance of the visit.
• GAO may not visit every project included in the Quicklook; projects not visited will receive written questions based on the content of their monthly review packages.
• PMs are invited to bring any charts or personnel that they need to help answer the questions; however, GAO prefers to have an informal discussion – NOT a lecture.
• When preparing responses, keep in mind that Quicklook is about cost growth against the Agency’s MPAR Baseline; responses should keep in mind the full mission cost – including launch vehicle and UFE
• Refer questions about things you are not responsible for to correct organization– LV questions should generally be directed to LSP– MD-held UFE questions should be directed to the MD, although it is ok to state the value
• When in doubt, it is ok to take an action to provide a response after the visit.
1/22/2012 17
This is the project manager’s opportunity to add context and explain all the good work their team is doing.
2-Page Project Summary Anatomy
1/22/2012 18
Project Challenges Overview
Project Essentials
Contract Info
Project Performance Summary
Project Timeline Project Update
Project Office Comments
Draft 2-Page Project SummariesWhat to look for
• Overview and Project Essentials – Is the information correct?
• Project Update and Summary– Does the GAO accurately portray the status?– Are there any errors in the analysis?
• Project Challenges– GAO’s challenge categories are fixed and will
not change– Assertion of a challenge should be supported
by the analysis described in the Project Update
• Contract Info, Project Performance, and Project Timeline – no action from PM required; Procurement and
OCFO will review to ensure consistency with other external reporting
How to Respond1. Project comments
– Will be printed in the “Project Office Comments” section of the final report
– Limited to 500 characters (appr 100 words)2. General Comments
– Focus on errors in understanding or interpretation of information previously provided to GAO as well as points of disagreement with the GAO’s analysis.
– These comments will not appear in the report but may help drive the discussion with the GAO about how they can revise
3. Technical Comments– Comments provided in this section will be used
to correct any factual errors in the report. – Corrections must be accompanied by evidence
1/22/2012 19
GAO will transmit draft summaries directly to project managers with response requested in 5 business days; Response must go through full agency review process in this time.
WHAT DID THE DATA SAY?So we gave GAO all this data…
1/22/2012 20
GAO Quicklook – Mar 2011• Maintained recommendation for GAO’s “Business Case Approach,” but recognized
improvements due to JCL policy• Found average cost growth of 14.6% and schedule growth of 8 mo; nearly all growth
attributed to projects baselined before 2009• Main Observations:
– Technology Challenges: • NASA begins development with insufficient technology maturity• Evidence: Majority of projects passed PDR with critical technologies at TRL < 6
– Design Challenges: • Projects pass CDR with unstable design• Evidence: Nearly all projects passed CDR with <90% releasable drawings
• Additional Observations– Funding Challenges: JWST, Ares, Orion funding “did not align with plan”– Launch Vehicle Challenges: mainly medium class – addressed more fully in another GAO report– Development Partner Challenges– Parts Challenges – addressed more fully in GAO Quality Parts Audit– Contractor Management Challenges
• Report stated that GAO will conduct a more thorough analysis of NASA’s EVM data in future audits and that [cost] transparency is insufficient during formulation1/22/2012 21
22
Mar 2011 QLB - Cost Growth
NASA agrees with GAO’s cost and schedule growth figures
The majority of NASA’s cost and schedule growth is attributed to projects baselined prior to 2009
1/22/2012
Funding Issues Launch Issues Parts Issues Technology Issues
Design Issues Contractor Issues
Development Partner Issues
0
2
4
6
8
10
122009
2010
2011
# Pr
ojec
ts w
ith
GA
O-id
entifi
ed Is
sue
Challenges SummaryQuicklooks published in 2009, 2010, 2011
Subjects of other GAO audits1/22/2012 23
Data Provided During 2011
Technology Maturity and Drawing Release
• Self-reported data from DCIs submitted to GAO appear to support the hypothesis that immature technologies at PDR leads to cost growth for NASA missions
– The distribution does not support a strong correlation
– JWST and MSL drive the correlation, but have cost growth clearly attributable to other factors.
• The data shows low correlation between releasable drawings at CDR and cost growth
1/22/2012 24
MSL did not provide drawing data; MAVEN is excluded from CDR analysis because CDR was not complete at time of DCI submission
0% 5% 10% 15% 20% 25% 30%-10.0%
10.0%
30.0%
50.0%
70.0%
90.0%
110.0%
130.0%
150.0%
Class A
Class B
Class C
Class D
% Immature Technologies at PDR
Perc
enta
ge D
evel
opm
ent C
ost
Gro
wth MSL
NPP
JWST
0.00%
20.00%
40.00%
60.00%
80.00%
100.00%
120.00%-30.0%
-10.0%
10.0%
30.0%
50.0%
70.0%
90.0%
110.0%
130.0%
150.0%
% releaseable drawings at CDR
Perc
enta
ge c
ost g
row
th
NPP
GPM
SOFIA
Juno
JWST
GAO hypothesis
Data Provided During 2011
Drawing Release
CDR Now# Releasable A C
Total # anticipated B D
NASA: % releasable = A/BGAO: % releasable = A/D
NASA’s 90% guidance is based on what is known at the time
• There appears to be some disagreement between how NASA and GAO calculate the percent of releasable drawings
• NASA calculates the percentage of releasable drawings at CDR based upon the total number of drawings anticipated as of CDR
• Drawing growth after CDR typically occurs as a result of risk realization; it is impossible to predict which risks will come to pass and which of those will require drawing changes
• By NASA’s calculation, several projects did meet the best practice guideline of 90% releasable drawings at CDR
Aquarius GPM GRAIL Juno JWST LADEE LDCM MMS MSL NPP OCO-2 RBSP SOFIA TDRS0.00%
10.00%20.00%30.00%40.00%50.00%60.00%70.00%80.00%90.00%
100.00%
% of Drawings Releasable at CDR
GAO Calculation Assumed NASA calculation
1/22/2012 25
Data Provided During 2011
Software
• 13 projects provided complete software data– 4 projects - Aquarius, MSL, NPP, and JWST - reported growth in TLOC greater than 25%– 5 projects reported a decrease in TLOC since PDR – Juno, LDCM, RBSP, SOFIA, TDRS
• After removing projects whose cost growth is clearly attributed to other factors, the mathematical correlation between development cost growth in growth in TLOC is high (.91) but the lack of significant distribution does not support a relationship between cost growth and software growth
26
-40% -20% 0% 20% 40% 60% 80% 100% 120%-20.0%0.0%
20.0%40.0%60.0%80.0%
100.0%120.0%140.0%160.0%
NPPSOFIA
JWST
GPM
LDCM
MSL and Aquarius Excluded
% Growth in TLOC, PDR to Current
% D
evel
opm
ent C
ost G
row
th
-100% 0% 100% 200% 300% 400% 500% 600% 700%-20.0%0.0%
20.0%40.0%60.0%80.0%
100.0%120.0%140.0%160.0%
NPPSOFIA Aquarius
MSL
JWST
GPM
All Quicklook Projects
% Growth in TLOC, PDR to Current% D
evel
opm
ent C
ost G
row
th
1/22/2012
AGENCY MESSAGESWhat is Headquarters saying to GAO?
1/22/2012 27
Formulation• Formulation is an iterative
process that aims to balance requirements and resources.
• At KDP-C the Decision Authority agrees that the design is sufficiently mature and the requirements match the resources
• The confirmation decision results in an agreed-to cost and schedule baseline; budget is committed to the project as part of the PPBE process
• The balance of requirements and resources is continuously assessed throughout a project’s lifecycle
ResourcesRequiremen
ts
Changes in schedule or projected LCC during formulation should not be treated as slips or cost growth; rather, this is a normal part of the formulation process.1/22/2012 28
Road to Confirmation(For SMD Projects to date)
PDR
SRB C&S (JCL) Assessment
SRB Technical Assessment
Program Office Recommendation
Division (Theme) Recommendation
Center Recommendation
ICEs or ICAs1
Project Report
Project Response to SRB
DPMC APMCCMC
SRB C&S (JCL) Assessment
SRB Technical Assessment
SMD Recommendation
ICEs or ICAs1
Project Report
Project Response to SRB
OCFO/SID Analyst Assessment
SRB C&S (JCL) Assessment
SRB Technical Assessment
Project Report
Project Response to SRB
ICEs or ICAs1
Approval to Proceed (Cat 2&3)2
Approval to Proceed
(Cat 1)
The Confirmation decision takes into account many inputs. There is no “right answer”
JCL JCL JCL
1. when available2. Cat 2 and 3 missions do not require an APMC, but Cat 1 missions do. With 7120.5E, the AA will approve entry to Implementation for all projects
1/22/2012 29
Center Recommendation
JCLs and UFE• The use of JCL in decision making represents a shift in the
paradigm of how project cost estimates, budgets and schedules are established.
• This method of estimation moves beyond prior practices of calculating a “point estimate” and adding an additional level of reserves.
• Because the UFE is a product of the probabilistic basis of estimates, any reduction to the UFE will reduce the probability of achieving the project cost and schedule targets
• Usage of UFE should not be interpreted as a “funding issue;” the UFE is part of the full amount that NASA budgets for a mission.
• The term “UFE” should be used rather than “reserves” to denote the more rigorous cost estimating policy NASA now uses
1/22/2012 30
With the release of 7120.5E, NASA no longer uses the term “reserves.” PMs should update charts to use the term
“Project-held UFE.”
Management vs Agency CommitmentsCost Schedule
Management Agreement• Managed by PM• Based upon Project’s
own Cost Estimate• Should be at 50% JCL or
greater
Agency Baseline Commitment• Based upon JCL and other
inputs• At 70% JCL or as
determined by the DA• Difference between ABC
and MA is budgeted as MD-held UFE
Agency Baseline Commitment• Based upon JCL and other
inputs• At 70% JCL or as
determined by the DA• Difference between ABC
and MA is additional schedule margin held by the Agency
Project-held UFE
MD-Held UFE
Project Schedule
Margin
Agency Schedule
Margin
Management Agreement Managed by PM
Based upon Schedule developed by PM
Should be at 50% JCL or greater
In some cases, the Decision Authority deems it prudent for the Agency to hold additional schedule margin above the project’s schedule. In these cases, the Management Schedule should not be viewed as acceleration of the Agency’s schedule
Not to scale
1/22/2012 31
What happens when funds are not appropriated or performance is poor?
o When requested funds are not appropriated, or performance is poor, work is pushed to future years, and budget requirements increase in the out years.
o This may lead to cost growth and operating plan changes.
o If reductions happen in subsequent years, the profile becomes unsustainable. FY10 FY11 FY12 FY13 FY14
400
450
500
550
600
650
700Notional Example
FY10 Budget request
FY10 Appropriation + FY11 request
FY11 Appropriation + FY12 request
FY12 Appropriation + FY13 request
1/22/2012 32
EVMTotal Project (Combined In- and Out- of House)
IIn-House
Out-of-House
ContractsPartnersCP & FPI
>$50MCP & FPI >$20M
CP & FPI <$20M FFP
7 Principles
Certified System
32 Guidelines NA NA NA
7 Principles applied across all elements of Total Project
• NASA’s projects are accomplished through combined efforts of suppliers and NASA in-house efforts– Total In-House with targeted procurements– Prime contract with government providing Level-Of-Effort (LOE) oversight– Many unique mixtures of the above two scenarios
• NASA applies EVM to contractors at the project level and requirements are flowed down to subcontractors
NASA projects tend to be complex; EVM is not necessarily applied to all suppliers but it is addressed at
the total project level.
EVM is one of many project management tools that NASA uses. NASA recognizes EVM implementation as an area of concern and is committed to improvement in this area
1/22/2012 33
Technical Leading Indicators
• NASA is enhancing the ability to monitor and assess the stability of programs and projects as well as the maturity of their products through the development process
• Comprehensive 3-Step Approach1. The Entrance and Success Criteria for Lifecycle reviews are being
enhanced to further account for product maturity• These will be documented in the Rev B update to NPR 7123.1
2. A required set of 3 parameters that will address product stability and maturity will be added as a NID to NPR 7123.1A• Mass Margins• Power Margins• Review item closures
3. A set of recommended Common Leading Indicators is being developed• Added to NPR 7120.5E Formulation Agreement and Program/Project Plan
templates
1/22/2012 34
Any Questions?
1/22/2012 35