Amer.tahani

HeadquartersIndependent Program & Cost Evaluation

State-of-the-Art in

Independent Review Execution

Tahani Amer and Kaiser Adeni Review Managers

Independent Program Assessment Office

8th Annual 2011 NASA Program Management Challenge

February 10, 2011

HeadquartersIndependent Program and Cost Evaluation

Outline

• Look Back at FY10 Reviews• Review Process Improvements• SRB Balance & Structure • SRB Members Roles & Responsibilities • SRB Coordination (between Mission Directorates,

Centers, and Programs/projects (P/p))• SRB Products• SRB – HB updates• IPAO State-of-the-Art Activities• Look Forward

2


Look Back at FY10 Reviews

• 8 Programs and 20 project review activities executed in FY10: Preliminary Design Review: MAVEN Critical Design Reviews: GRAIL, RBSP, GPM, TDRS, LDCM, MMS,

OCO-II System Integration Reviews: MSL, GRAIL and Juno ORR/FRR/PLAR: WISE Program Approval Review: RPS and LQP Program Implementation Review: SCaN, ESMP, D/NF Special Reviews: MSL, CxP, Aquarius, SOFIA 8 Non-IPAO reviews: 2- GOES-R, ASP, AvSP, ATP, FAP, ENAS, CAS

3


FY10 Completed Reviews

4


FY10 Completed Reviews

5


FY10 IPAO Completed ReviewsIPAO supported 8 non-IPAO review activities in FY10

6


Review Process Improvements

• SRB process improvements: • Quick Look Reports (one-pagers);• 30-day reporting requirement;• Increased coordination for programmatic assessments;• Readiness-to-proceed assessments;• Alternate opinions for non-consensus boards;• Key Decision Point (KDP) Decision Memo improvements;• Deferral of program reviews approved at APMC;• Institutionalized electronic signature of SRB approval letters;• Streamlining of ToR content as defined in draft NPR 7120.5E;

7


Review Process Improvements

• SRB process improvements (continued): • Better coordination of SRB team nomination process; early

stakeholders involvement in the process to ensure proper team balance of competency/currency/independence;

• Final reports are posted on APMC electronic repository (https://nx.arc.nasa.gov/nx/dsweb/View/Collection-93608); notification at APMC when reports are posted;

• Strengthening and improving the rigor and integration of the technical and programmatic assessments;

• Working with SMD and the Centers to formulate principles on conducting joint Program/project reviews when there is a significant external partner involved. Efforts are evolving w/ExoMars and JPSS.

8

https://nx.arc.nasa.gov/nx/dsweb/View/Collection-93608


SRB Balance & Structure

• SRB members should be competent, current, and independent from the management or advocacy chain of the P/p, with membership balanced between the host Center and other organizations to ensure the needs of the convening authorities are met.

• Although balance of each SRB member is important, ultimately the goal is to have the SRB, as a whole, balanced.

• More inclusive set of discussion with Center and TA • It is not a numerical formula, but it goals to meet

Agency’s goals

9


Competency Relevant experience and expertise with the technical, specific

technologies and programmatic domain of the P/p under review. Currency

Addresses recent/current experience and expertise in programmatics or the technical domain(s) of the project or program under review.

Understanding of current Agency governance, project management and systems engineering policies, procedures and methods, specifically NPR 7120.5, NPR 7123.1, NPD 1000.5, specific Center practices and procedures and the SRB Handbook.

Independence Not in the programmatic chain of command of the program or

project and have no conflicts of interest either personally, institutionally or organizationally.

10

SRB Balance & Structure


SRB Members Roles & Responsibilities

• A member of an SRB is an “agent” of the convening authorities: More emphasis on programmatic risk assessment and analysis;

More emphasis on the individual member independent report (IMIR);

Advise the SRB Chair and the Review Manager (RM) on areas that require particular attention by the SRB per their area of expertise;

Support the SRB “Quick-Look Report” summaries and briefings to the Convening Authorities (CA), and provide expert opinion to SRB preliminary and final reports.

11


SRB Coordination

• Coordination with the Mission Directorates: Early and frequent coordination is conducted with the Mission Directorate Program

Executive (PE) to: Establish SRB membership

Chair with support from RM establishes the discipline areas to be covered PE/MDs provide a list of potential candidates. CAs approve or provide alternate

nominations RM leads the vetting effort for Organizational and Personal Conflict of Interest

(OCI/PCI) in coordination with the LaRC Legal Office and Contracting Officer(s) Finalize Terms of Reference (ToR) content

Content is reviewed and agreed-to prior to final approval by the CA Identify any additional Mission Directorate review criteria

Establish timing of the Site Visit Establish post-review out brief schedule

CMC/DPMC briefing APMC briefing (Cat 1 and some Cat 2 Programs/projects) APMC special topics for Cat 2 project life cycle reviews

Pause and Learn (PAL)

12


SRB Coordination

• Coordination with the Center Coordination with the Center is conducted to:

Coordinate and develop a list of SRB candidates from the Center based on the discipline areas established by the Chair

Facilitate the review of the ToR content and ensure that the Center specific requirements are being met

Golden Rules JPL Rules

Facilitate Center CA approvals of ToRs, team nominations, etc. Establish dates for post-review CMC briefing

Establish a Community of Practice (CoP) at each Center (in progress) Knowledge sharing of best practices and expectations for independent assessment Review manifest coordination and approval Identify Center personnel to participate in SRBs Assist with nomination of high potential candidates for RM assignment with IPAO Request potential Center Review Managers as detailees to IPAO Develop a CoP for the Agency after completion on the discussion with Centers

13


SRB Coordination

• Coordination with Program/project: Program/project pre-planning coordination begins ~120 days prior to Site Visit to:

Establish Site Visit review requirements Discipline areas to be covered

Data drop timeline for cost and schedule products Establish agreement on Terms of Reference (ToR) review criteria

Single ToR for LCR (0.5E) Readiness to Proceed Review Establish timing of the Site Visit Establish post-review out brief schedule

Quick- Look Report (One Pagers) P/p briefing CMC briefing DPMC briefing APMC briefing (Cat 1 and some Cat 2 P/p)

14


SRB Products

SRB products produced for each Site Visit: Briefings

One-pager (Quick Look Report) P/p Briefing CMC Briefing DPMC APMC Briefing (if a Cat 1 Project)

ToRs, team nomination letters, alternate opinionsVetting Package

OCI/PCI mitigation plans/annual vetting SRB member resumes/bios SRB approval letters

Reports Final SRB summary report to include each SRB individual reports

to DA. https://nx.arc.nasa.gov/nx/dsweb/View/Collection-93608 IPAO Review Record RRD for the Agency

15

https://nx.arc.nasa.gov/nx/dsweb/View/Collection-93608


SRB –HB Update

Guidance to the P/p and SRB members

The SRB-HB is posted on the http://nodis.hq.nasa.gov/policy_letters/NM_7120-81_.pdf

Updates to comply with NPR 7120.5E

16

http://nodis.hq.nasa.gov/policy_letters/NM_7120-81_.pdf




IPAO State-of-the-Art Activities

17

• JCL assessments and improved programmatic analysis: Using Defense Contract Management Agency (DCMA) and Government

Accounting Office (GAO) criterion as standard evaluation criteria for schedules

Implementation of standard analysis timeline allowing for time to work with the projects to improve cost and schedule concerns

Analysis processes have been defined and documented in Standard Operating Procedure Instructions (SOPI)

• Better risk assessments: Using state-of-the-art tools for integrated risk analysis Establishment of Schedule Analysis Working Group (SAWG) to develop a

CoP for programmatic risk analysis

• Forensics Study: Analysis of the SRB findings Identify trends and systematic issues Provide recommendation to improve PM at NASA


IPAO State-of-the-Art Activities

• Lessons Learned (LL) after each review IPAO members share the LL with RMs and PAG Analysts Database of LL

• ILCR - Customer/CA/SRB member surveys Developed ILCR surveys & Approved by CA Implementing the survey: RBSP

• Training: RMs & Chairs• Outreach Effort: Articles, PM Track, Visits to Centers, PAL

• Developmental Program: Detail opportunity to support Agency Project Management

18


Look Forward

• Implementation of the NPR 7120.5E• Update the SRB Hand Book• Continue coordination with MDs, Centers, and P/p• Communities of Practice• SRB Balance• Independent Reviews involving external partners• Independent Reviews and Technology P/p

19


Back-up Slides

20


One Step PDRLife Cycle Review Overview

KDP-C KDP-C

PDR Readiness Assessment PDR-LCR

Technical Baseline with C/S/R and Integrated Assessment of Technical and Programmatic

Baseline

KDP-B KDP-B

Periodic SRB Involvement as Appropriate

P/pBrief

CenterBrief

MDBrief

Not To Scale

(30 Days)

Quick Look Report

(30- 90 days)

-Required prior to LCR-Report to DA for life cycle reviews preceding KDP B&C and during any major replan or rebaseline (3)

Programmatic Data Drops to SRB (includes JCL Model)

Deliveries start at 100 days before site review

(2)

FOOTNOTES:1. A One Step Review may be used for any LCR.2. Appendix I provides information on the readiness assessment, quick-look reports and checkpoints associated with life cycle reviews.3. For all other life cycle reviews, report to Chief Engineer if significant unresolvable disagreements.

(2)

CheckPoint if needed.(2)

21


Two Step PDRLife Cycle Review Overview

Resolve Tech Issues/Risks, Update

Cost/Schedule Baseline

Independent Integrated PDR Assessment

PDR Readiness Assessment PDR

-Required prior to LCR-Report to DA for life cycle reviews preceding KDP B&C and during any major replan or rebaseline (3)

Technical Baseline with Cost,

Schedule, and Risk Information

Integrated Assessment of Technical and Programmatic

Baseline

KDP-B KDP-B

Periodic SRB Involvement as Appropriate

P/pBrief

CenterBrief

MDBrief

Not To Scale

(1-6 months) (30 Days)

Quick Look Report

(30-90 days)

PDR LCRKDP-C KDP-C

Quick Look Report

Programmatic Data Drops to SRB (includes JCL Model)

(2)

(2)(2)

FOOTNOTES:1. A Two Step Review may be used for any LCR2. Appendix I provides information on the readiness assessment, quick-look reports and checkpoints associated with life cycle reviews3. For all other life cycle reviews report to Chief Engineer if significant unresolvable disagreements

CheckPoint if needed.(2)

22

Amer.tahani

Technology

Transcript of Amer.tahani