NA
SA M
SFC
Vict
ory
Sol
utio
ns
SB Diversity
MIPSS Team
PCEC v2.2 Overview
2017 NASA Cost & Schedule Symposium
29 Aug 2017
Victory Solutions MIPSS Team
Outline
• PCEC v2.2 New Features
• Robotic Spacecraft Model Updates for v2.2
• Crewed & Space Transportation Systems (CASTS) Updates for v2.2
• REDSTAR Overview
• Future Work & Analyses
2
Victory Solutions MIPSS Team
PCEC Overview
• PCEC is a parametric cost model developed and maintained by NASA
beginning in late 2013. It comprises the PCEC Interface and PCEC Library
– The PCEC Library is a collection of cost estimating artifacts used to create an
estimate of a space flight hardware system
– The PCEC Interface is an Excel Add-in with buttons to access artifacts from the
PCEC Library for building an estimate in Excel
• PCEC is currently best suited for estimating the cost to design, develop, and
produce the following types of space systems
– Spacecraft: Earth Orbiting Satellites, Planetary Probes, Rovers
– Launch Vehicles: Multi-stage rockets, liquid and solid engines
– Human Space Flight Systems: Crew Capsules, Orbiters, Habitats
• PCEC is available to the General Public via the NASA Software Catalog
and to NASA Civil Servants via the ONCE Database
3
PCEC V2.2 NEW FEATURES
PCEC v2.2 Overview
4
Victory Solutions MIPSS Team
PCEC Status Update
– Big increase in users earlier this year due to media campaign
associated with the 2017/2018 NASA Software Catalog release
– Continue to add a couple of users each month
• PCEC v2.2 will be available this week
– Announcement and download link will be sent out to existing users
5
• PCEC v2.1 is the
current active
release (Aug 2016)
• Current user
counts as of mid-
Aug 2017
– 350+ Users
– 45 Countries
Victory Solutions MIPSS Team
Summary of Major Changes in
PCEC v2.2
• Usability Improvements
– Launch Resume Capability
– Post-Launch Worksheet Linking
• Additional Phase E/Operations Estimating Options
– Mission Operations & Data Analysis (MO&DA) CERs
– Linking to Operations Cost Model (OCM) and Space Operations Cost
Model (SOCM) estimates
• CER Importer
• Other Improvements
– Compatibility with Mac Office
– CER Updates
– Template Updates
– Many under-the-hood enhancements, bug fixes, and corrections
6
Victory Solutions MIPSS Team
PCEC v2.2
Launch Updates (1 of 2)
7
The Launch Resume
capability allows the
user to pick up the
editing of a Launch-
created estimate
where they left off
Launch now has two paths:
• Start New Estimate
• Resume Existing Estimate
Victory Solutions MIPSS Team
PCEC v2.2
Launch Updates (2 of 2)
• The Launch Resume feature currently only works with estimates
created with v2.2
• Other Launch updates in v2.2
– MOCET, OCM, and SOCM-linked elements can be added to the WBS
– New indicators for WBS line items
– Options for formatting templates
8
The Resume routine
reads WBS metadata
(CERs selected, options
for linked models, etc.)
and enables the same
WBS building as the
initial run of the LaunchThe routine updates
the estimate file
based on the
additions / deletions /
edits made during the
Resume session
Victory Solutions MIPSS Team
PCEC v2.2
Post-Launch Sheet Linking
9
New routines automate the
linking of a template added
after the Launch run to other
key worksheets in the estimate
• Link to Globals: Links all Global variables on the active sheet to
the Globals worksheet
• Link to Summary: Inserts a new line into the WBS (location
indicated by the user), and the active sheet is linked to it
• Link to Subsystem Inputs: Inserts a section on the Subsystem
Inputs worksheet for entering input data for the subsystem
Victory Solutions MIPSS Team
PCEC v2.2
MO&DA CERs
The Mission Operations
& Data Analysis
(MO&DA) CERs provide
an alternative option for
estimating Phase E costs
for robotic spacecraft
missions, both Near Earth
and Planetary
10
MO&DA templates
have the same look
as other templates
and allow for
conducting
sensitivity analysis
and modeling
uncertainty
A separate SOCM Score
Calculator worksheet is included
with the MO&DA template to
centralize input of data for use
throughout the estimate
Victory Solutions MIPSS Team
PCEC v2.2
Linking to OCM and SOCM
OCM and SOCM are not included
with the PCEC distribution but can
be requested for NASA use
– Distribution rules/process TBD
11
Both the Space Operations
Cost Model (SOCM) and
Operations Cost Model (OCM)
are now supported for
inclusion into a PCEC estimate
Estimate results are
pasted or linked into a
template for including in
the WBS like the models
PCEC already links to
(NICM and MOCET)
Victory Solutions MIPSS Team
PCEC v2.2
CER Importer
Current Limitations:
– Only supports linear or power CERs
– Any independent variable used, if not in PCEC
already, must be added manually to the Interface
12
Data must first be input on
the pre-formatted CER
worksheet (included) so it
can be read by the Interface
The user
selects the
workbook
containing the
documentation
and the CERs
to be imported
The CER(s) can then
be imported into the
user’s local copy of
the Interface for use
throughout the tool
PCEC now makes it
easier to bring in a user’s
custom CERs for use in
an estimate
Victory Solutions MIPSS Team
PCEC v2.2
Other Improvements
• Mac Compatibility of PCEC Interface
– Now runs on Office for Mac 2016
• Worksheet Template Updates
– First Pound Cost Templates
– MOS/GDS Development
– SOCM Score Calculator
– General template for user-added CERs
• Bug Fixes / Corrections
– LV and Crewed Templates
– Userform Search bug
– Running Launch multiple times
– Correlation matrices for v1 CERs
– Unit space CV calculation for all CERs
• Under-the-hood enhancements for maintainability and adding new
estimating capabilities13
ROBOTIC SPACECRAFT
MODEL UPDATES
PCEC v2.2 Overview
14
Victory Solutions MIPSS Team
Robotic SC Model Updates
Summary
Mission Operations & Data Analysis (MO&DA) and Other CER Updates
• MO&DA Objective/Approach
• MO&DA Input Collection
• MO&DA Data Analysis
• MO&DA (Phase E) CER development
• Other CER Updates
15
Victory Solutions MIPSS Team
PCA
Other Data
Analysis Methods
Constructive Processes
Acceptable CER
Performance?
Yes
No
Mocet Inputs
SOCM Inputs
PCEC Inputs
Key Driver
Input SetsSupports
Regression Based
CERs
CER
Performance
AnalysisAttempt to Identify
Common Attributes
to Explain
Error/Residuals
Multiple Regression
PCEC CERs
Candidate CERsNear Earth Prime Mission
Planetary – Cruise
Planetary - Encounter
GOAL: Derive Phase E (MO&DA) CERs for PCEC Robotic SC
leveraging the MOS/GDS Development effort
New Input CandidatesDifferent Input Combinations;
Data Collection of Additional Info;
Combination Inputs (SOCM
Scores)
PCEC MO&DA Analysis
Development Approach
16
Victory Solutions MIPSS Team
PCEC MO&DA Analysis
NASA WBS Elements Included
• The PCEC MO&DA analysis
includes elements of Mission
Operation Systems (MOS) and
Ground Data Systems (GDS)
from WBS 7.0 & 9.0 during
Phase E.
NASA WBS items
included for this
effort in Phase E
17
Victory Solutions MIPSS Team
PCEC CADRe Data Normalization
Current Project Data Set
• Phase E data for the 42 missions
shown here were normalized
• The set covers recent missions
and includes representatives from
each NASA science discipline
• The normalization process shows
the traceability to the official
CADRe data
– All assumptions and changes have
been documented
• The normalized data for each of
these missions has been provided
to the lead organizations for their
review
18
Victory Solutions MIPSS Team
INPUT CANDIDATES
PCEC MO&DA Analysis
Initial Input Candidates
• Multiple information sources have been reviewed to generate the
initial input candidate list:
– CADRe: Fields in Part B (technical)
– Cost Models: Space Operations Cost Model (SOCM), PCEC
Normalized Data Library, MOCET
– Over 150 input candidates were identified for each mission
19
Victory Solutions MIPSS Team
PCEC MO&DA AnalysisSearch for New Input Candidates
• Assessments have been made to identify whether a single variable exists to provide a
‘reasonable’ estimate
– Many candidates have been explored without showing any clear trend
– Similar result to pre-SOCM efforts in the 1990’s
• Explored potential to build on SOCM analyses (by using intermediate SOCM scores
or other lower-level SOCM inputs as predictor variable)
20
Victory Solutions MIPSS Team
PCEC MO&DA AnalysisRationale for Planetary/Near-Earth Split
• Initially, a CER was developed using the entire data set which
consists of both Planetary and Near Earth missions. The CERs
created from this initial effort produced less than desirable results.
• Ultimately, the best results were obtained by splitting the data set
into Near Earth and Planetary missions.
– The planetary data was then split into cruise and encounter phases to
capture the unique attributes of these operating phases.
• This split makes sense when you consider the unique aspects of
operating in the Near Earth environment when compared to deep
space operations.
– Near Earth missions have many options in terms communications
networks (NEN, TDRSS). Planetary missions rely on the Deep Space
Network (DSN).
– Near-Earth missions often employ high-heritage s/c and MOS/GDS
elements. Planetary s/c are typically customized to minimize mass and
tailored to their application.21
Victory Solutions MIPSS Team
PCEC MO&DA Analysis
Leveraging of SOCM
• Even with the split in the data set, reasonable CERs still could not
be identified
• Several Space Operations Cost Model (SOCM) inputs were added
to the candidate input set
– SOCM was originally developed by a team of operations experts and
cost analysts
– SOCM Level 1 Overall Score and Payload Score were added; The
SOCM Level 1 Scores have remained unchanged over the last 2
updates (2000 & 2004); The Overall Score supported the Near Earth
CER and the Payload Score supported the Planetary CER
– Parametric performance statistics significantly improved with the added
SOCM Score inputs
• SOCM has been tested with all the PCEC Missions and shows
reasonable results
– SOCM estimates did better for Near Earth Primary Mission and
Planetary Encounter than for Planetary Cruise
22
Victory Solutions MIPSS Team
PCEC MO&DA Analysis
SOCM Level 1 Score
• The SOCM Level 1 Scores for Near Earth and Planetary
applications are a composite of several input parameters
– Level 1 Score consists of two equations that determine level of effort of
the science and engineering teams. E.g., the relevant input parameters
needed to calculate the Near Earth score include:
23
Victory Solutions MIPSS Team
• SOCM estimates were developed for each of the PCEC missions
– Approach leverages expert judgement from operations specialists which results in ‘intuitive’
changes from inputs (most inputs are ‘causal’ vs ‘associative’)
– Used Level 1 inputs and a ‘select set’ of Level 2 inputs (5-6 inputs)
– Results appear reasonable
• Some correlation of the SOCM Level 1 score to MO&DA cost requirements appears
to be demonstrated by this analysis
– Results are used to identify ‘Combination Inputs’ to carry into the PCA/Regression
24
PCEC MO&DA Analysis
SOCM Performance Testing
Victory Solutions MIPSS Team
PCEC MO&DA Analysis
CER Results
• Equations take the form of:– $k/month = (Parameter 1)α · (Parameter 2)β ·…….. · EXP(Constant)
• SOCM Level 1 Score input requires a separate calculator in PCEC
25
Victory Solutions MIPSS Team
PCEC MO&DA AnalysisModeling Performance Comparisons
Mean Error = 4%
Mean Abs. Error =
23%
Standard Dev. = 26%
Mean Error = 4%
Mean Abs. Error = 27%
Standard Dev. = 33%
NEAR EARTH PRIMARY MISSION
PLANETARY ENCOUNTER
26
Victory Solutions MIPSS Team
MO&DA Phase E
Planetary Cruise Calculator
• Regression of the planetary cruise data did not result in an
acceptable CER
• This is likely due to the fact that the activities during cruise vary
drastically from mission to mission and the time to accomplish these
activities varies greatly depending on the time required to reach
each planetary destination.
• During the data analysis effort, we discovered that encounter cost is
a fairly good predictor of cruise cost
27
Victory Solutions MIPSS Team
% D
iffe
ren
ce =
(E
sti
mate
–A
ctu
al
)/A
ctu
al
PCEC MO&DA AnalysisOverall MO&DA Error by Mission
• Derived MO&DA prediction CERs have reasonable performance.
• Overall prediction error is roughly +/- 50% which is a slight improvement
over the SOCM approach.
• Near Earth CER performs slightly better than the Planetary Encounter CER
28
Victory Solutions MIPSS Team
PCEC CER Updates
• As PCEC continues to develop into a full lifecycle cost model and is
used more and more to support various NASA costing activities,
opportunities arise for the PCEC developers to improve the CERs in
the model.
• PCEC is a constantly evolving model that relies on user feedback
and continued expansion of the normalized data set.
• Thank you for providing your inputs – please continue to do so. They
will ultimately make PCEC a better model.
• The following charts describe the CER changes that have resulted
from this iterative community process and will be included in the
upcoming PCEC v2.2 release.
29
Victory Solutions MIPSS Team
PCEC MOS/GDS Development
Updated CER Results
• Equations take the form of:
– $k/month = (Parameter 1)α · (Parameter 2)β ·…….. · EXP(Constant)
• Updated SOCM Level 1 Score in MO&DA development resulted in an update to the MOS/GDS Development CERs
30
Victory Solutions MIPSS Team
Mean Error = 6%
Mean Abs. Error = 30%
Standard Dev. = 35%
Mean Error = 4%
Mean Abs. Error = 23%
Standard Dev. = 28%
NEAR EARTH MISSION
PLANETARY MISSION
PCEC MOS/GDS Development Performance Comparisons – Updated CER
31
Victory Solutions MIPSS Team
PCEC Robotic SC SubsystemsComm. (SSPA Planetary) - Updated CER
• Equations take the form of:– $k = (Parameter 1)α · (Parameter 2)β ·…….. · EXP(Constant)
• Given the small number of planetary S/C using SSPAs in the dataset, there was concern that the previous CER was somewhat oversubscribed.
• In re-examining the data, we found that SSPA mass was the best predictor of cost with a very clear distinction between Mars operating SSPAs and the rest of the planetary SSPA data set.
32
Victory Solutions MIPSS Team
PCEC Robotic SC Subsystems
C&DH NRC/RC - Updated CER
• Equations take the form of:– $k = (Parameter 1)α · (Parameter 2)β ·…….. · EXP(Constant)
• Concern was expressed that PCEC C&DH CERs for Recurring Cost (RC) and Non-Recurring Cost (NRC) did not fully capture the complexity of various C&DH implementations.
• In re-examining the data, Non-Recurring and Recurring costs were recombined in order to enable development of a reasonable CER. An updated parameter (# of Boards) was vetted and introduced in to CER process.
33
Victory Solutions MIPSS Team
PCEC Robotic SC SubsystemsStructures & Mech. (RC) – Updated CER
• Equations take the form of:– $k = (Parameter 1)α · (Parameter 2)β ·…….. · EXP(Constant)
• Mission Risk Class was behaving in a counterintuitive manner. Variable was removed with negligible impact on CER performance.
34
Victory Solutions MIPSS Team
PCEC Robotic SC Subsystems
ACS (NRC) – Updated CER
• Equations take the form of:– $k = (Parameter 1)α · (Parameter 2)β ·…….. · EXP(Constant)
• Year of Launch input variable was behaving in a counterintuitive manner. Variable was removed with negligible impact on CER performance.
35
Victory Solutions MIPSS Team
PCEC Robotic SC Subsystems
ACS (RC) – Updated CER
• Equations take the form of:– $k = (Parameter 1)α · (Parameter 2)β ·…….. · EXP(Constant)
• Year of Launch input variable was behaving in a counterintuitive manner. Variable was removed with negligible impact on CER performance.
36
Victory Solutions MIPSS Team
Summary & Findings
• The current PCEC approach has eliminated need for “wrap factors”
across all NASA WBS items except the Science Team
• Prior operations modeling efforts (SOCM) included participation from
operations experts; Current approach is attempting to build on the
insights from past efforts by statistically correlating identified
operations cost drivers to documented costs
• Use of the SOCM Level 1 scores strongly improved parametric
performance for MO&DA and MOS/GDS Development CERs
– The SOCM Level 1 score combines inputs characterizing all elements
and was developed by an integrated team of operations experts and
cost analysts
37
CASTS MODEL UPDATES
PCEC v2.2 Overview
38
Victory Solutions MIPSS Team
CASTS UPDATES
in PCEC v2.2
• Expansion of Full Life Cycle Cost Capability: 2 updates
1. Facilities, Launch and Flight Operations add-ins
• Operations Cost Model (OCM)
2. Prototype multi-system integrated LCC capability
• Multiple vehicles and/or vehicle configurations over multiple years
• Flight-rate-sensitive CER’s
– Fixed and Variable cost as function of flight/production rate
– Cash flow (budget) per year: Spread & Non-spread cost
• Updated CERs
– Minor updates to 8 CERs
– Additional reference data points added to historical data for multiple CERs
• Documentation
– Updated CASTS User’s Guide for v2.2 to be available October
– Virtual Black Books
• PCEC v2.1 Cost Data Sheets completed and uploaded to REDSTAR for all systems included
in CASTS CERs; Version 2.2 updates in work
• REDSTAR Resource Data and system Technical Data sheets completed for Saturn and
Apollo elements (S1C, SII, SIVB, CSM, LM), Skylab Airlock and OWS, Shuttle Orbiter
39
Victory Solutions MIPSS Team
Expanded CASTS LCC
Capability & Context
40
Acquisition• $ = f(Indep Var, CPLX, AF, …)
• Indep Var = wt, thrust, SLOC, etc
Outputs:
• DDT&E by WBS element
• TFU by WBS element
Facilities• $ = f(Indep Var, CPLX1, CPLX2, …)
• Indep Var = Footprint, GLOW, thrust
Outputs:
• $ per Facility
Operations• $ = f(Indep Var, ratios, CPLX1, CPLX2, …)
• Indep Var = Mission characteristics, system
element definitions, propellant load, etc.
Outputs:
• CER’s: $/Year as f(flight rate)• $ = (Var CPF x Flt Rt) + Fixed CPY
• $ = [A x Cum Flts^b x Flts/Yr^c] x
Flts/Year
CASTS OCM
Mission Model
• Flights/Year by System Element & Market Segment: 1) NASA Manned – ISS 2) Unmanned USG 3)Unmanned Commercial
DDTE
Facilities
Production
Recur Ops
DDTE• DDT&E Cost Elements
spread by year
Facilities• Facility Cost by facility
as needed – f(capacity,
flight rate)
Production• Reusable Hardware by
element as needed –
f(capacity, flight rate)
Recurring Operations• Recurring Cost by WBS
element– f(flight rate)
System Definition• Sketch Drawing
• Subsystem Weights & Descriptions
• Mission Profile
• Ops Flow
Integrated Model - LCC/Yearly Cash Flow
Victory Solutions MIPSS Team
Updated CASTS WBS
• Updated CASTS Work Breakdown Structure
– Addition of Facilities, Launch, and Flight Operations cost elements
– Appears in the Launch Vehicle and Crewed WBSs
41
Program Segment Vehicle Segment (cont'd) Vehicle Segment (cont'd)
Program Mgt & Support Thermal Protection Software SegmentSystems Engr & Integ Passive Flight Software
Vehicle Segment Propulsion Ground Software
Integration, Ass'y, Checkout Liquid Engines Test SegmentCrew Structures Solid Motors System Test Operations
Thrust Structure Reaction Ctl/Orb Maneuv Sys System Test Hardware
Adapters Avionics & Power Ground SegmentSecondary/Support Structs Guidance, Nav, & Control Ground/Test Support Equip
Tanks Telemetry & Tracking Tooling
Intertank Command, Ctl, Data Handling Facilities
Mechanisms Range Safety/Flt Termination Launch Operations
Thrust Vector/Flight Control Electric Power Flight Operations
Separation Shroud/Fairing
Recovery Crew Systems
Other
Main Propulsion Systems
Victory Solutions MIPSS Team
Expanded CASTS WBS
42
• Expanded WBS elements for Facilities, Launch, and Flight Operations
Ground Segment Ground Segment (cont'd) Ground Segment (cont'd)Facilities Launch Operations Flight Operations
1 Launch Pad L1 Vehicle Processing F1 Flight Planning
2 Vertical Processing Facility L2 Processing Engineering F2 Mission Software
3 Horizontal Processing Facility L3 Recovery Operations F3 Simulation & Training
4 Launch/Mission Control Center L4 Program Mgt & Support F4 Mission Control O&M
5 Payload Processing Facility L5 Facility O&M F5 reserved
6 Mobile Launch Platform L6 Base Support F6 Payload Analytical Integ
7 Landing Facility L7 Propellants F7 Crew Operations
8 Base Infrastructure L8 GSE Spares F8 Program Mgt & Support
L9 Ground Software O&M F9 reserved
L10 Payload Processing F10 Network Support
Taken together CASTS now provides capability to estimate all elements
and phases of crew and space transportation systems Life Cycle Cost.
Victory Solutions MIPSS Team
REDSTAR Overview
43
• REDSTAR is the REsource Data STorage And
Retrieval System managed by the NASA/Marshall
Space Flight Center's Engineering Cost Office.
– Established in 1971
– NASA-wide library of cost, programmatic, and
technical data pertaining to the space program
• REDSTAR is both a physical library and an online
searchable database maintained in SharePoint
– Accessible by NASA Civil Servants and Support
Contractors
– Request access via NAMS (nams.nasa.gov)
• REDSTAR contains over 40,000 documents of
which over 12,500 are available electronically. The
data spans from the total program level down to the
subcomponent level for over a 150 space flight
systems dating back to the 1910’s.
For questions about REDSTAR or for research support,
contact Mary Ellen Harris, REDSTAR Librarian
[email protected] Document History
Victory Solutions MIPSS Team
Future Work & Summary
• Ongoing / Future Work
– Milestone Estimating Capability (Robotic SC)
• Currently exploring at correlations between cost inputs at the start of Phase
B, project changes, and Actual Costs and/or Reserve requirements
– CER Library Selection
– Advanced Life Cycle & Architecture Modeling
– PCEC Website Update
• PCEC v2.2 Summary
– Available this week to existing users / requesters
– Usability improvements should make it easier to build and update PCEC
estimates in the future
– More life cycle estimating capabilities will provide the ability to estimate
through the end of the mission
44
PCEC Email Contact: [email protected]
Application Website: https://software.nasa.gov/ , search for PCEC
Victory Solutions MIPSS Team
We always welcome feedback on the
tool / analyses and suggestions on
future capabilities. Find us at the
Symposium or send us an e-mail.
Contact Info
45
Brian Alford
Booz Allen Hamilton
Mark Jacobs
TGS Consultants
Shawn Hayes
TGS Consultants
Richard Webb
KAR Enterprises
Gideon Bass
Booz Allen Hamilton
BACKUP
46
Victory Solutions MIPSS Team
Principal Component Analysis Approach
1) A correlation matrix was generated
to get a sense of the of the
dependency between variables.
• Several of the variables appeared to be
correlated, making PCA an attractive
method to apply to the data set.
2) The principal components were
determined using an algorithm
developed in Python.
• The first 6 principal components which
account for 85% of variance in the data set
were selected and used to determine which
of the 20 variables were most likely related
to cost.
3) For each of the 21 data sets examined, 4 subsets of the 20
variables were run through a multiple regression routine to
determine the new cost estimating relationships. 47
Victory Solutions MIPSS Team
Process
PCA Results
Run RegressionMinimal Model
48
Victory Solutions MIPSS Team
NASA Space Missions
Modeling Lessons Learned
• PCA can help identify a manageable subset of potential costing inputs that are
the main contributors to cost differences from a much larger candidate set
• A consistent approach for data normalization is essential; Programmatic
differences between the projects can strongly influence official costs
– PCEC normalization adjusts the data to a defined set of rules/procedures
• Do not trust regression results without a thorough sanity check
– Often, “associative” instead of “causal” inputs can yield counter-intuitive results (that may be
misdirected); Best approach maximizes utilization of available “causal” inputs
– It is important to understand reasons for outliers, which can lead to model enhancements
• A combination of PCA, regression, and constructive modelling approaches
appears to offer many benefits over reliance on a single technique
– Enhances flexibility to capture unique aspects associated with NASA robotic science
missions
– Adjustments to regression results need to be supported by data analysis
• Accuracy of technical and cost data should always be reviewed and questioned
– differences often exist in assumptions behind different values for the same
item from different sources
49
Top Related