2021 NASA Cost and Schedule SymposiumVirtual Event

April 7, 2021

DISCLAIMERThe cost information contained in this document is of a budgetary and planning nature and is intended for informational purposesonly. It does not constitute a commitment on the part of NASA, JPL, and/or Caltech.

© 2021 California Institute of Technology. Government sponsorship acknowledged.

Software Cost Estimation for COVID

Sherry StukesJet Propulsion Laboratory, California

Institute of Technology4800 Oak Grove DrivePasadena, CA 91109

sherry.a.stukes@jpl.nasa.gov

Georgia BajjaliehLogical-R Joint Venture, LLC

NASA Ames Research CenterMoffett Field CA 94035

georgia.bajjalieh@nasa.gov

Contents

vBackground and OverviewvHistorical DatavModels and MethodologiesvFindingsvSummary

2

Background and Overview

3

Background - VIPERvVIPER rover mission to the

moon’s south polev100 earth-days

vMission to find water icevWhere is water ice located? vWhere are the highest

concentrations?vHow deep is the ice?

vFirst U.S. rover launched to the lunar surface since the Apollo missions in the early 1970’s

4

Reference: VIPER home page, https://www.nasa.gov/viper

*VIPER in Formulation Phase when the Rover Software Cost Estimate was conducted in 2020

Key Cost Estimating Challenges (1)

5

v Parametric models are not designed to capture unique aspects of VIPER Rover Software v VIPER is being developed as a research and technology project,

rather than as a space flight programv VIPER is a risk tolerant mission similar to NASA “Class D”

payloads v Key portions of the rover’s software are being designed and

implemented as ground software

v Reflecting affects of tailored NASA Procedural Requirement (NPR) requirements

v No direct Lunar Rover analogiesv Draw on bits and pieces from prior projectsv Look for systems as similar as possible

Key Cost Estimating Challenges (2)

6

v Integration of large open source software productsv RGSW – Gazebo (3D dynamic simulator)v RSIM – ROS2 (Robotic Operating System stereo image

processing and tools)

v Impact of COVIDv Models do not directly capture pandemic cost impactv Agile development may be less efficient as Agile

principles promote face-to-face meetings and a highly collaborative environment

v Personnel productivity may be different than expected

Rover Flight Software Architecture

7

v3 CSCIsvRover Ground Software (RGSW)vRover Flight Software (RFSW)vRover SIMulations Software (RSIM)

vDecomposed to module level for software lines of code sizing

vPlanned for high software reuse vIntegration large open source

modules

Historical Data

8

Historical Data – Resources

9

v Reviewed Resource Prospector BoE document dated 12/19/2013v Limited information and details for five BoEsv Researched updated information

v Data Sourcesv JPL SMART (Software Measures and Analysis Reporting

Tool) – JPL proprietaryv CADRe (Cost Analysis Data Requirement) – available on

ONCE site https://oncedata.hq.nasa.gov/Main.aspxv ASCoT (Analogy Software Cost Tool) – also available on

ONCE sitev NASA projects

v Used “JPL Mission Software: State of Software Report 2018” as a reference for software size

10

Historical Data Collection Project Data

Project Center Source Why Relevant

LADEE ARC Reviewed CADRe and ASCoT data with Software Project Manger for LADEE Lunar orbiter, Class D mission

LCROSS ARC Only sizing data available Robotic lunar impactor, low-cost

LRO GSFC ASCoT data only Robotic lunar orbiter, 3D mapping

GRAIL JPL Data from CADRe, ASCoT, and SMART (used for RFSW and RSIM) Lunar orbiter and simulation software

MPF JPL Data from CADRe Rover, tech demo, all new software code

MER JPL Data from CADRe, ASCoT, and SMART Rover

MSL JPL Data from CADRe, ASCoT, and SMART Rover

M2020 JPL Preliminary software size data from Project Rover, subset analogous to RGSW

GRO GSFC SMART data only Simulation software

STS Centaur GRC SMART data only Simulation software

STS 51E and 61I JSC SMART data only Simulation software

11

Historical Data CollectionWhat We Found

v Software database contentvVariety of sourcesvAcross NASA CentersvOrganized by element

v Established NASA sourcesv Easily accessible

VIPER rover software elements compare favorably with historical data.

Historical Flight Software Data Analogies

Reference: JPL State of Software Report, 2018

VIPER RFSW size is in family with JPL historical projects.

12

Models and Methodologies

13

SEER-SEM and COCOMO are in wide use across NASA.

Models and Methodologies

v Parametric Modelsv Primary – SEER-SEMvDeveloped by Galorath IncorporatedvV8.3vWidely used within NASA

v Cross-check – COCOMO (SCAT)vJPL version of COCOMO IIvModel implemented in Excel and enhanced to capture

uncertainty with Monte Carlo add-invAvailable to NASA Centers from JPL

v Analogy Software Cost Tool – (ASCoT)v Rules of Thumb (SLOC/WM)v Compare and contrast cost estimates

14

SEER-SEM DetailsInput Data Summary

vSLOC (Logical)vNewvReused “as is”vReused

modifiedv % re-designv % re-codev % re-test

vTotal deliveredvEquivalent new

vKnowledge BasesvEstablishes

default parameter settings

vBased on industry averages

vModified for VIPER

SoftwareElement Platform Application Acquistion

MethodDevelopment

MethodDevelopment

StandardRFSW Unmanned Space Flight Systems New & Reused Agile - Novice CommercialRGSW Unmanned Space Flight Systems New & Reused Agile - Novice CommercialRSIM Unmanned Space Simulation New & Reused Agile - Novice Commercial

Rover Software Knowledge Base Selections

Notes Logical SLOC

Terms

ESLOC - Equivalent (new) Source Lines of Code

RSIM - Rover Simulation Software

SLOC - Source Lines of Code

RFSW - Rover Flight Software RGSW - Rover Ground Software

ProjectsLADEE - Lunar Atmosphere Dust and Environment ExplorerRP -Resource ProspectorOSRF - Open Source Robotics Foundation

SW Element

Reuse “as is”

Reuse Modified

% Redesign

% Recode

Reuse Source

New SLOC

Total SLOC

ESLOC SLOC

Units SLOC SLOC % % Project SLOC SLOC SLOCRGSW 26371 2860 25 50 RP 10638 40490 17256RFSW 55858 85885 7 26 LADEE 53530 196096 89218RSIM 303700 21600 15 15 RP, OSRF 26320 351930 30852

Rover Software Size Summary

15

SEER-SEM Details Software Sizing

vSample of software sizingvSized by modulevConducted software reuse studyvCounted with PySLiC

repo/folder Description CategoryRework Design

Rework Code

Retest Code Language

Logical SLOC Inherited

Soure Project

Logical SLOC Expected Notes Modified

rockCraterGen rock and crater generation Legacy 15% 30% 50% Matlab 1000 RP 1200 1000craterDisplay crater marking GUI Legacy 30% 30% 20% C++ 1200 RP 2200 1200moonShine synthetic DEM generator Legacy 10% 20% 40% C++ 6400 RP 7000 6400craterProfile improve crater profile New C++ 500

tile_to_gazebotile to Gazebo model processing Legacy 50% 40% 20% Python 650 RP 800 650

NSS protocol New C++ 300RP NSS/NIRVSS Sim behavior Legacy 15% 50% 10% Python 1000 RP 2000 1000

interface to external components New C++ 1000 high uncertaintyfault ui New C++ 1500WheelSlipPlugin Legacy 10% 10% 100% C++ 400 OSRF 400 A lot of testing and tweaking will be done on this plugin400SDF parameters manual process a LOT of manual effort goes into thisCAD to URDF manual process New XML 2000 a LOT of manual effort goes into thisRover Appearance shaders for rover model New GLSL 400 Effort under repres entedvipersim_tools util scripts and tools Legacy 10% 30% 30% C++ 1000 RP 2000 1000Gazebo compositor rewrite Legacy 0.10% 0.250% 1% C++ 284500 OSRF 284500

sdformat Legacy 0.00% 0.00% 0.00% C++ 32300 OSRF 32300deleted from total, reuse as is for Gazebo

Inherited Expected New Code325300 351620 26320 21600 303700

Modified As Is

*Note: These are a subset of modules so do not sum to the totals. 16

SEER-SEM DetailsKnowledge Base (KB) Adjustments

v7 categories, 37 parameters

vSet at Least, Likely, and Most to create a distribution

vCOCOMO parameters set independently

17

18

= Knowledge Base parameter settings changed due to COVID impact*Approximate cost Impact is to the SEER-SEM knowledge Base settings for the individual parameters

SEER-SEM DetailsSEER-SEM COVID Parameter Adjustments

Least Likely Most Least Likely Most Least Likely Most

Low Low+ Nom+ Low Nom- Nom+ Low Low+ Nom+ ~8%

Slight penalty to reflect the difficulty of team member coordination of requirements analysis and design work resulting in miscommunication of information.

Low+ Nom- Hi Low+ Nom- Hi Low+ Nom- Hi 3%Slight penalty to reflect reduced programmer interaction in the working environment.

Nom Nom+ Hi- Nom Nom+ Hi- Nom Nom+ Hi- ~2%

Represents the Ames software development organization with slight penalty for reduced capabilities due to COVID. Coordinated with the VIPER Mobility software leads.

Nom Hi Hi Nom Hi Hi Nom Hi Hi+ ~5%Increased Likely to account for being geographically distributed. RSIM Most could be worse than RGSW and RFSW.

Low Nom- Nom Low- Low+ Nom- Low Low+ Nom- ~6%Lowered slightly to account for reduced access to servers.

Resource and Support Location

Nom+ Hi- Vhi Nom+ Hi- Vhi Nom+ Hi Vhi 9%Increased slightly, especially on Most setting, to account for longer wait times for support. RSIM could be more severly impacted.

Parameter DefinitionRGSW RFSW RSIM Cost

Impact*Notes

white cells =KB

Analyst Capability

Overall capability of the software analysts assigned to this development, in terms of their ability to function effectively as a team. Rate the team, not individuals.

Programmer Capability

Development Practices Use

Multi Site

Resource Dedication

Average capability of programmers assigned to the development; includes factors related to capability, motivation and working environment.

Rates the development organization's use of software engineering processes and methods that are considered to be "best practices".

Organizational and site diversity of the project team. This parameter should consider circumstances that prevent project team members from being able to collaborate on the spur of the moment. Consider physical location, time zone differences, etc.

Availability of the host and target machines to the development organization.

Diffuculty of access to development resources and support. Physical and political separation must be considered.

Analogy Software Cost Tool (ASCoT)

NASA ASCoT provides Cluster Analysis and KNN for early lifecycle analysis. ASCoT is available in ONCE web site.

The NASA Analogy Software Cost Tool (ASCoT) is a web-based tool that provides a suite of estimation tools to support early lifecycle NASA Flight Software analysis.

vClustering - Analogy estimation method that estimates software development effort using a variation of principle components to derive the clusters

vKNN - Analogy estimation method using three nearest neighbors algorithm

19

Analogy Software Cost Tool Results

KNN and Cluster analysis compare favorably with VIPER estimate.

VIPER EstimatesInputs:Mission Size: SmallMission Type: RoverDestination: Inner PlanetaryNumber of Instruments: 4Number of Deployables: 3Inheritance: HighRedundancy: Single String

VIPERRFSW +RGSW

Estimate

KNN Effort: Redundancy = Single String Nearest Neighbors Neighbor's Effort Neighbor's Distance to VIPEREstimate: MPF 1080 1.08997 WMs Grail 868 1.1

DS1 1042.8 1.12

KNN SLOC: Redundancy = Single String Nearest Neighbors Neighbor's SLOC Neighbor's Distance to VIPER Delivered Estimate: MPF 94.3 0.78 236103 LOC LRO 62.9 1.07 ESLOC

Grail 156.1 1.07 106

Cluster Effort: Redundancy = Single String Nearest Neighbors Neighbor's Effort Neighbor's Distance to VIPEREstimate: Grail 868 1.1619 WMs NuStar 493 1.12

LADEE 492 1.12

619 WM

619 WM

Cluster placement: Cluster 1GLORYGPM CoreGrailLADEENuStarOCOWISE

20

Findings

21

Estimate Results Summary

Software effort from models appears to be reasonable with KNN and Cluster results

vEffort estimates for software development only

vSEER-SEM and COCOMO estimates at 50% confidence

vLower than the KNN

vNear and within the Cluster range

vReflects COVID impact

SEER-SEM ASCoT ASCoTEstimate

(WM)Cluster (WM)

Cluster Range (WM)

RGSW 80 78 89RFSW 587 519 600Substotal 667 597 689 997 868 - 1080 619 492 - 868RSIM 266 270 290 - - - -

ElementAggressiveCOCOMO

Estimate (WM)

ConservativeCOCOMO

Estimate (WM)

ASCoTKNN Range

(WM)

ASCoTKNN

(WM)

Note: WM – Work Months

22

v SEER-SEMv Based on adjusted

KBs v Accounts for COVID

v COCOMO conservative estimate assumes v Possibility that a

larger amount of new and modified code will be required

v Higher volatilityv Required Reliability is

Software Class B-v COCOMO estimates

schedule to be between 20-26 Months

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

20 40 60 80 100 120 140

Like

lihoo

d of

Occ

urre

nce

Effort (Work Months)

VIPER Rover RGSW Development Effort CDF (Requirements through SW I&T)

Recommend Budget be between a 50th with reserves and 70th without reserves

COCOMO Optimistic78-95 WMs

COCOMO Conservative 89-108 WMs

SEER 80-98 WMs

RGSW Development Effort CDF

23

Using the Estimate

v The rover software estimate modeled without COVID impacts compared favorably to the project’s internal rover software grassroots estimate

v The COVID impacts modeled resulted in a cost increase of ~6% to the project’s rover software estimate

v Monitor rover software actuals against the estimate for variances regularly to detect cost or schedule threats

v Refine the estimate based on results of variance analysis or changing conditions

24

Summary

25

Lessons Learned – User Perspectivev Advantages of parametric models

v Model parameters provide structure to think about and tune for outliers or “black swan" events (e.g., COVID)

v Greater flexibility, conducive to scenario analysis/what-if analysis

v Use a combination of methodologies and tools to exploit strengths of each when building a life-cycle cost estimate

v Cost estimating in a rapidly changing environment will require more frequent updates to the estimates and cost models

26

Lessons Learned – Estimator Perspectivev Virtual software cost estimates can be donev Communication with the estimating team

v Conduct frequent, regular meetingsv WebEx helps keep team focusedv Keep all members of the team involved

v Current software cost estimating models may be able to capture the impacts of COVIDv Appears possible that existing parametric model parameters

can be used as proxies for COVID effectsv Still need to validate / ground truth estimate with actualsv Need to determine if there are COVID effects that need to be

modeled (long-term stress, “staycation” vs. vacation, etc)

v We need to prepare for the next “black swan” event and adapt the models accordingly

27

v The VIPER Rover software estimate was challenging for multiple reasons, particularly in terms of modeling COVID effects

v The impact of COVID needs continued studying and may be useful for estimating cost of future “black swan” events

v Assembled a knowledgeable virtual team representing the major project functions

v Invested time for software sizing and paid attention to reused code and incorporated large open source modules that required adaptation, integration, and testing

v Identified a specific set of cost parameters in the parametric estimating models with settings that were sensitive to a COVID environment

v Established a benchmark at PDR that can be updated throughout the life of the project

Conclusions

28

Definitions• ARC – Ames Research Center• ASCoT – Analogy Software Cost Tool

• BoE – Basis of Estimate• CADRe – Cost Analysis Data Requirement• CCSDS – Consultative Committee for

Space Data Systems• CDF – Cumulative Distribution Function

(“S” curve)• cFS – Core Flight Software

• C&DH – Command and Data Handling• COCOMO – COnstructive COst Model

• COVID– COronaVIrus Disease• CSCI – Computer Software Configuration

Item• DoD – Department of Defense• DS1 – Deep Space 1

• EDL – Entry, Descent, Landing

• ESLOC – Equivalent (new) Source Lines of Code

• FY – Fiscal Year• GDS – Ground Data System• GRAIL - Gravity Recovery and

Interior Laboratory

• GRO – Gamma Ray Observatory• ICE – Independent Cost Estimate

• JPL – Jet Propulsion Laboratory• KB – Knowledge Base• KNN – Nearest Neighbors

• KSLOC –1,000 Source Lines of Code• LADEE – Lunar Atmosphere and Dust

Environment Explorer• LCC – Life Cycle Cost

29

Definitions• LCROSS – Lunar Crater Observation and

Sensing Satellite• LRO – Lunar Reconnaissance Orbiter• M2020 – Mars 2020• MER – Mars Exploration Rover• MOS – Mission Operations Support• MPF – Mars Pathfinder• MSL – Mars Science Laboratory• NASA – National Aeronautics and Space

Administration• NPR – NASA Procedural Requirement • NuSTAR – Nuclear Spectroscopic

Telescope Array• ONCE – One NASA Cost Engineering• OSRF – Open Source Robotics Foundation• PDR – Preliminary Design Review• PySLiC – Python Software Line Counter

• RFSW – Rover Flight Software• RGSW – Rover Ground Software• ROS2 – Robot Operating System2• RP – Resource Prospector• RSIM – Rover SIMulations Software• RSW – Rover Software• SCAT – Software Cost Analysis Tool• SEER-SEM – Software Evaluation and

Estimation of Resources-Software Estimating Model

• SLOC – Source Lines of Code• SMART – Software Measures and

Analysis Reporting Tool• SQA – Software Quality Assurance• STS – Space Transportation System• VIPER – Volatiles Investigating Polar

Exploration Rover• WM – Work Month

30

ReferencesArtist’s Impression of VIPER on the Lunar Surface, VIPER home page, https://www.nasa.gov/viper.Jorgensen, M. and Grimstad, S., Over-optimism in software development projects: “The Winner’s Curse”, 15thInternational Conference on Electronics, Communications and Computers, ISBN: 0-7695-2283-1, IEEE, 2005.

NASA Procedural Requirement (NPR) 7120.8A, NASAResearch and Technology Program and Project ManagementRequirements”, https://nodis3.gsfc.nasa.gov/, September 14, 2018.

NASA Procedural Requirement (NPR) 7120.5E, NASA Space Flight Program and Project ManagementRequirements”, https://nodis3.gsfc.nasa.gov/, Change Number 18, March 12, 2020.

Taber, William, Python Source Line Counter (PySLiC) User Guide, JPL, Release Version 1.0, May 22, 2019.NASA Ames Research Center, Resource Prospector Mission Rover Analysis of Alternatives Flight Software CostEstimate, Software Basis of Estimate Review, December 19, 2013.

Johnson, James, et al., One NASA Cost Engineering Database (ONCE) Overview and Update, 2019 NASA Cost andSchedule Symposium, Houston, TX, August 2019.

Hihn, Jairus, et al., ASCoT, the NASAAnalogy Software Cost Tool Suite: Expanding Our Estimation Horizons, 2018IEEE Aerospace Conference, Big Sky, MT, March 2018.

One NASACost Engineering (ONCE) Database, available at: https://oncedata.hq.nasa.gov/Main.aspxSoftware Measures andAnalysis Reporting Tool (SMART) repository, JPL, 2020.Stukes, Sherry, et al., JPLMission Software: State of Software Report 2018, JPL, September 2019.System Evaluation and Estimation of Resources - Software Estimating Model (SEER-SEM), Galorath Incorporated,Version 8.3, August 27, 2020.

Lum, Karen, Software Cost Analysis Tool Users Document, JPL D-26304, Version 1.2, October 13, 2005.

31