Complex Decision-Making Applicationsfor the NASA Space Launch System

Garry LylesNASA Marshall Space Flight Center

Huntsville, AL 35812Garry.Lyles@nasa.gov

Tim FloresNASA Marshall Space Flight Center

Huntsville, AL 35812Timothy.J.Flores@nasa.gov

Jason HundleyZero Point Frontiers Corp.

Huntsville, AL 35801256-503-7527

Jason.P.Hundley@nasa.gov

Stuart FeldmanZero Point Frontiers Corp.

Huntsville, AL 35801256-694-0240

Stuart.M.Feldman@nasa.gov

Timothy MonkZero Point Frontiers Corp.

Huntsville, AL 35801334-524-7379

Timothy.S.Monk@nasa.gov

Abstract—The Space Shuttle program is ending and elementsof the Constellation Program are either being cancelled ortransitioned to new NASA exploration endeavors. The NationalAeronautics and Space Administration (NASA) has workeddiligently to select an optimum configuration for the SpaceLaunch System (SLS), a heavy lift vehicle that will provide thefoundation for future beyond low earth orbit (LEO) large-scalemissions for the next several decades. Thus, multiple questionsmust be addressed: Which heavy lift vehicle will best allowthe agency to achieve mission objectives in the most affordableand reliable manner? Which heavy lift vehicle will allow for asufficiently flexible exploration campaign of the solar system?Which heavy lift vehicle configuration will allow for minimizingrisk in design, test, build and operations? Which heavy liftvehicle configuration will be sustainable in changing politicalenvironments?

Seeking to address these questions drove the development of anSLS decision-making framework. From Fall 2010 until Spring2011, this framework was formulated, tested, fully documented,and applied to multiple SLS vehicle concepts at NASA fromprevious exploration architecture studies. This was a multistepprocess that involved performing figure of merit (FOM)-basedassessments, creating Pass/Fail gates based on draft thresholdrequirements, performing a margin-based assessment with sup-porting statistical analyses, and performing sensitivity analysison each. This paper discusses the various methods of thisprocess that allowed for competing concepts to be comparedacross a variety of launch vehicle metrics. The end result wasthe identification of SLS launch vehicle candidates that couldsuccessfully meet the threshold requirements in support of theSLS Mission Concept Review (MCR) milestone.

TABLE OF CONTENTS

1 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 APPROACH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 FOM-BASED ASSESSMENT . . . . . . . . . . . . . . . . . . . . . . 24 DRAFT SLS REQUIREMENTS ASSESSMENT . . . . 45 AGGREGATE MARGIN STUDY WITH SUPPORT-

ING STATISTICAL ANALYSIS . . . . . . . . . . . . . . . . . . . . 46 CONCLUSION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

ACKNOWLEDGMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . 5BIOGRAPHY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

978-1-4577-0557-1/12/$26.00 c©2012 IEEE.1 IEEEAC Paper #1278, Version 1, Updated 10/17/2011.

1. INTRODUCTIONThe decision for a new heavy-lift launch vehicle is rooted inthe 2005 Exploration Systems Architecture Study (ESAS),which reinvigorated serious interest in a national heavy-liftcapability. The Constellation Program (CxP), formed inresponse to the ESAS, established long duration explorationmissions to the Moon and eventually Mars, which involvedlaunching high-volume, large mass payloads into low earthorbit (LEO). The Ares V launch vehicle served the heavylift need of the CxP and underwent multiple design conceptstudies and design analysis cycles between 2007 and 2010.

As CxP was recommended for cancellation in early 2010,the National Aeronautics and Space Administration (NASA)conducted several internal studies on heavy-lift vehicle archi-tectures capable of performing a wide range of missions atreduced cost and schedule. The studies, most notably theHeavy-Lift Launch Vehicle (HLLV) study and the Heavy-Lift Propulsion Technology (HLPT) study, identified severalfamilies of launch vehicles whose features were distinguishedby the number of stages, types of solid and liquid propulsionsystems, and outer mold line (OML) design characteristics.These families included both side-mount and in-line config-urations, 27.6-foot and 33-foot OML diameters, 1.5 thru 3stages, multiple booster options, and a wide range of liquidpropulsion engines and solid rocket motor options.

The NASA Authorization Act of 2010 was passed by boththe U.S. House of Representatives and U.S. Senate and wassigned by the President on October 11, 2010. This Actdirected NASA to develop a Space Launch System (SLS)capable of delivering crew and heavy cargo to LEO andbeyond. As its initial capability, the SLS must be capable oflifting 70 metric tons of payload to LEO. A full capability of130 metric tons is achieved with the addition of a combinedUpper and Earth Departure Stage to the initial configuration.

With national stakeholder needs passed into law and variousvehicle options to chose from, NASA George C. MarshallSpace Flight Center (MSFC) established a team to developa decision-making framework in which to communicate ef-fectively with agency decision-makers the characteristics andrelative merits of the likely vehicle candidates. This frame-work will be summarized in this paper, with a specific focuson the methods themselves rather than the results produced.

1

2. APPROACHThe team set out to establish a process by which the SLSSteering Committee could identify vehicle candidates thatcould meet documented NASA Mission Concept Review(MCR) success criteria. The process involved establishinga standardized set of process inputs, each of which havetheir own documented set of groundrules and assumptions.These standardized input sets and associated analytical pro-cesses were in the areas of performance, cost, and reliabilityestimation. By forming a common set of groundrules andassumptions for all discipline areas and associated teams, aconsistent set of vehicle data was produced that would allowfor relative comparisons to be made across many vehicleconcepts.

As with most complex problems, it is very difficult to definea single methodology or process that will identify a clearlyattractive candidate from a multitude of options. Therefore,the decision team set out to define a multifactored processthat would allow the problem to be assessed from a varietyof angles. This process (as shown in Figure 1) includes thefollowing steps:

FOM-Based Assessment with figure of merit (FOM)s andcriteria established from stakeholder metrics in the areas ofperformance, schedule, programmatics, and affordability.Draft SLS Level 1 Requirements assessment with all con-cepts evaluated on a pass or fail basis against the minimumthreshold values across a variety of metrics.Aggregate Margin Assessment which compares all vehicleconcepts to the draft Level 1 requirements and their marginwith respect to each. This also includes supporting statisticalanalysis on the same metrics.

3. FOM-BASED ASSESSMENTStakeholders in the SLS Decision

In the heavy-lift vehicle decision process, the early estab-lishment of an agreed upon set of FOMs derived from thestakeholders in the decision is imperative. In this manner, thestakeholders are widely represented in the decision-makingprocess, and the resulting vehicle decision is based on thisrepresentation rather than other external factors. Further-more, it provides a standard method for measuring andcomparing competing concepts rather than independentlyassessing and potentially eliminating competitors without alarger-view of the trade environment.

For NASA, the decision making process is fairly straightfor-ward. The NASA Administrator, the Office of Science andTechnology Policy (OSTP), and the Office of Managementand Budget (OMB) provides input to the President on existingand potential NASA programs. The President proposes newprograms and/or the continuation of existing NASA programsin the annual Presidential Budget Request to Congress. TheCongress debates the merits of the proposed programs andeither chooses to authorize the programs or not. Theseauthorized programs are then funded through the appropri-ations process. In the simplest case, the conglomeration ofproposed, authorized, and appropriated programs are thenpassed and signed into law as the Nations Omnibus SpendingBill. This mutual agreement between the President and theCongressional Authorization and Appropriations processesare the general framework by which national decisions aremade.

Therefore, the Stakeholders for this decision are defined asthe President of the United States, both the U.S. House ofRepresentatives and U.S. Senate, and the NASA resourcestasked with carrying out the authorized, approved, and fundedprogram.

Stakeholder Derived Metrics

Determining the priorities of these stakeholders and the rela-tive importance of those priorities relative to one another is amore difficult task. The process used for this assessment wasas follows:

• Gather statements made by the stakeholders through theirrespective publications.• Bin those statements into general FOM categories.• Reduce similar statements into a single sub-FOM or whatcould potentially be termed heavy-life vehicle Needs, Goals,and Objectives (NGO).• Perform a first-order prioritization of those NGOs withinthe FOM categories.

The first step of this process is merely collecting statementsregarding the HLLV made by the President in the FY2011Presidential Budget Request and subsequent amendmentsthereof (speech at Kennedy Space Center (KSC) on April15th, 2010), as well as the Senate and House of Representa-tive Authorization Acts and Appropriations Reports. Thesestatements were documented as “Quotes” associated witheach source.

For instance, individual quotes from these sources at the timeof this study as it relates to the development schedule of theheavy-lift vehicle program are:

President’s Speech at Kennedy Space Center on April 15th,2010: “And we will finalize a rocket design no later than 2015and then begin to build it.”U.S. House of Representatives NASA Authorization Act(HR 5781): “...the Administrator shall strive to meet the goalof having the heavy lift launch vehicle authorized in thisparagraph available for operational missions by the end ofthe current decade”U.S. House of Representatives Appropriations Report: Notavailable at the time of this studyU.S. Senate NASA Authorization Act (S.3729): “...Priorityshould be placed on the core elements with the goal foroperational capability for the core elements not later thanDecember 31, 2016.”U.S. Senate Appropriations Report: “...The Committee in-vests in a new heavy lift rocket to be built by 2017, along withthe Orion capsule to carry astronauts, so NASA can againsend humans on new journeys of discovery.”

Organization and Simplification of Stakeholder Priorities

The compilation of statements by the stakeholders was thenorganized in matrix format based on the general FOM areasthat they targeted. These eventual general FOM areas foundwere Performance, Schedule, Programmatics, and Afford-ability aspects. Fortunately, several statements that wereextracted were closely related with slight nuances betweenthem. This led to a high level integration of closely relatedFOMs into what became known internally as “IntegratedFOMs”. These Integrated FOMs, if moved forward into theenvironment of a NASA program, may be considered draftNGOs for the new Heavy-Lift Vehicle program. That is,they are NGOs by which vehicle level requirements may bederived from (e.g. integrated vehicle shall deliver 70 metrictons to LEO).

2

Figure 1. Decision Analysis Process

Once consolidated into the “Integrated FOMs” or NGOs, ahigh-level ranking of the NGOs was performed in order toreduce the quantity even further. A simple High, Medium, orLow priority was assigned to each NGO with an associatedstakeholder that considers it to be of that priority. Thistechnique was used to eliminate NGOs from the assessmentwhich were medium or low in priority. This process resultedin all stakeholder priorities being distilled down into 16NGOs. Four of these were universally agreed to, three wereunique to the Presidential priorities, five were unique to theCongressional priorities, and four were additional derivedpriorities. Figure 2 lists these NGOs.

FOM Weightings

With the four FOMs (Performance, Schedule, Programmat-ics, and Affordability) and the criteria that composes theFOMs established (Figure 2), the team proceeded to weightthe FOMs. A commonly used technique to systematicallywork through complex decisions is the Analytical HierarchyProcess (AHP). This technique works by decomposing a par-ticular decision involving multiple variables into its individ-ual components and then comparing those two componentsindependently of all other considerations.

In this case, the individual FOMs are compared on a one-to-one basis to determine relative importance of one vs. theother. For instance, the four FOMs as listed are compared inan independent fashion systematically by answering six basicquestions:

• Is Performance more important than Affordability?• Is Performance more important than Schedule?• Is Performance more important than Programmatics?• Is Affordability more important than Schedule• Is Affordability more important than Programmatics?• Is Schedule more important than Programmatics?

These six questions are answered on a yes or no basis witha relative magnitude associated with that yes or no. Acomparative matrix is composed and once completed, allvalues are normalized within each FOM category so that a

Figure 2. Final Stakeholder Derived Priorities

relative comparison can be made between them. This resultsin an AHP factor that demonstrates the relative strength inscoring of a particular factor over all others. The AHP factoritself can be used directly as a weighting factor in mostcircumstances. An easier to interpret method is to derive thepercentage (out of a 100% total) that one factor holds over allothers. This is merely the percentage of the AHP factor thatone FOM accounts for divided by the total available AHP

3

factors. Through this AHP process, the FOM weightings forPerformance, Schedule, Programmatics, and Affordability arefound as shown in Figure 3 (based on a 100% total available):

Figure 3. AHP Determined FOM Weightings

NGO Scoring Criteria and Final Assessment

Once the weighting of each FOM is found, it is imperativeto set up a scoring method for each individual NGO. Inthis manner, data is measured using a consistently-appliedprocess across a variety of vehicle concepts. For this as-sessment, NGO definitions (for the NGOs shown in Figure2) were agreed upon and documented. However, each NGOrepresents a certain type of data. For instance, performancerepresents a calculated payload mass to a certain orbit.Cost metrics represents a cost estimate of some sort for agiven configuration, schedule and flight manifest. Schedulerepresents a development timeline found through a criticalpath assessment and so on. These aforementioned examplesrepresent standardized, quantifiable data sources. In thisassessment, there are also a multitude of data that representssubjective, hard-to-quantify NGOs. Examples of this datatype include workforce transition, life-cycle cost reductionchallenges, and others. Therefore, a standardized scoringmethodology is established for these particular attributes.

Additionally, it is required that all NGOs are measured ap-propriately. Two data conditioning steps must be undertakento ensure a representative outcome:

1) Ensure singular NGO quantities do not dominate the finalconsolidated measurement2) Ensure a large quantity of NGOs in a single FOM doesnot cause that FOM to dictate the results (regardless of theapplication of weighting factors)

Whereas some NGOs are on the scale of tens or hundreds,other NGOs are on the scale of billions. It is apparentthat a simple summation of these would result in the valuerepresented by the lower scale having very little impact inthe overall outcome. The method used to avoid this issueis a simple normalization between a fixed lower and upperboundary. For instance, performance is measured on the orderof metric tons. A given range for performance may be adesired performance between 70 to 130 metric tons. By fixinga common lower boundary at the minimum in the range anda common upper boundary at the maximum of the range, alladditional values can be interpolated between those two fixedpoints. If this process is repeated across all NGOs, then allare measured on a common scale.

The second data conditioning technique used for this assess-ment is averaging within a single FOM category to avoid thequantity of NGOs within a single FOM from dictating theoutcome. It is apparent that if all NGOs were merely summedfor the final results, Affordability would dominate the finaloutcome based on the quantity of Affordability metrics withinthe assessment. Therefore, before application of any weight-ing factors, all Affordability NGOs are summed and dividedby the quantity of Affordability NGOs (by stakeholder) to

produce an average Affordability subtotal (by stakeholder andtotal). An important point about this technique is that thisdictates that all NGOs are weighted equally within a singleFOM. Once these data conditioning steps are applied, vehicledata can be imported and scoring comparisons can be madeon any number of launch vehicle concepts.

4. DRAFT SLS REQUIREMENTS ASSESSMENTOne particular shortfall of the FOM Based Assessment is thatis does not capture how well the individual vehicles performagainst the draft SLS Level 1 requirements as establishedby the SLS Steering Committee in November 2010. Anassessment to these individual threshold values gives a muchbetter indication of which vehicle concepts have the ability tomeet standard NASA MCR success criteria based on eventualvehicle level requirements in all categories (cost, schedule,performance, reliability, etc.). In order to provide this insight,each individual requirement was given to the RequirementsAnalysis Cycle (RAC) teams as “Draft Threshold Require-ments” and “Draft Objective Requirements” across a varietyof metrics. All of these threshold values, if levied as programrequirements, must be met in order to be considered a feasiblevehicle concept for SLS MCR. With this “Pass or Fail”criteria established, the vehicle data can then be applied todetermine areas where the individual vehicle families fallshort of these minimal “Threshold” values. Alternatively,vehicle concepts meeting most or all of these “Threshold”values should be considered feasible vehicles for the purposesof a MCR. This process is further depicted in Figure 4.

Figure 4. MCR Feasible Vehicle Identification Process

5. AGGREGATE MARGIN STUDY WITHSUPPORTING STATISTICAL ANALYSIS

System Margin Assessment

At the inception of the SLS program, a measure of the“weighted system margin” is a very important metric thatshould be given appropriate thought when determining thepreferred vehicle approach. While it is very difficult toquantify certain subjective aspects of the impending vehicledecision, the quantitative categories are easily binned accord-ing the level of “goodness” that each concept provides. Thesenormalization bins give a relative measure of how muchmargin each vehicle concepts provides within the metricsestablished (with both the threshold and objective valuesconsidered). In addition, the weighting factors as currentlyunderstood can easily be applied to these quantitative metrics,resulting in a weighted system margin calculation.

4

Parametric Assessment of Vehicle FOMs

To support the assessment of the vehicle parameter space andto enable the incorporation of uncertainty and constraints inthe assessment methodology, a parametric and probabilisticMonte Carlo assessment of the RAC configurations was com-pleted. This assessment leveraged the work done by the RACcycle and applied uncertainty on each of the FOMs for eachof the vehicles. To complete the assessment, the followingcases were assessed:

1) Integrated vehicle assessment using multiple FOMweightings to assess sensitivity of the recommended optimalarchitecture to changes in FOM weighting2) Integrated vehicle assessment with sequential applicationof minimal requirementsa) Performance (to standard orbit)b) Safety and Reliability (LOC/LOM)c) Cost (total development, near term, and recurring cost)d) Schedule (first flight)

3) Integrated vehicle assessments with varying level of un-certainties on each of the FOMs

The assessments were completed using reference vehiclesthat were defined using the draft SLS Level 1 requirementsfrom the RAC process. Figure 5 depicts generic results ofthese Monte Carlo runs when implementing the Cost, Safety,and Performance gates. It can be interepreted as a probabilis-tic assessment of how many times out of all ran cases that thegiven configuration meets all levied requirement gates. Theresults from this assessment illustrates the sensitivity of the“optimal vehicle” to the applied constraints.

Figure 5. Parametric Assessment Sample Results

6. CONCLUSIONThe framework that was developed allowed for vehicle con-cepts and associated architectures to be compared on a varietyof levels. These include the ability of the configurations tomeet stakeholder needs, goals, and objectives, ability to meetSLS programmatic requirements (or requirement changes),and ability to provide margin to the program that will result inSLS having the ability to maintain both national and internalperformance, schedule, and affordability targets. Combined,this multifactored assessment selected vehicles that wouldhave the ability to successfully pass NASA MCR successcriteria. Furthermore, the output of the framework was vettedby agency and national leadership. In this manner, it provided

insight and confidence to agency decision-makers that theSLS concept chosen for further development will be a veryflexible and capable vehicle that should serve the needs of anation in its human space exploration endeavours for decadesto come.

ACKNOWLEDGMENTSThe authors would like to thank all those members of theSLS Steering Committee and the SLS RAC teams, as wellas those team members that contributed to the various studiesthat allowed for a solid comparison to be made among thevarious vehicle options.

The material in this paper is also based upon work supportedby NASA under Universities Space Research Association(USRA) award No. NNM08AA04A.

BIOGRAPHY[

Garry Lyles is the Space Launch Sys-tems Chief Engineer, leading the de-sign and development of the nations nextheavy-lift rocket for human explorationbeyond Earth orbit. Garry has a strongtechnical background and extensive ex-perience in technology programs andproject management. He led the SLSPlanning Teams architecture studies thatled to the current point of departure ve-

hicle now being further refined as the SLS Program ad-vances toward its System Requirements Review, scheduledfor Spring 2012. Previously, Garry was the Technical As-sociate Director of the Marshall Space Flight Centers Engi-neering Directorate, where he led systems engineering andintegration for a number of NASA programs and projects.Garry holds a bachelors degree in mechanical engineeringfrom the University of Alabama and he was inducted intoits class of Distinguished Engineering Fellows in 2010. Hehas been honored with awards such as the NASA Medalfor Exceptional Service, NASA Medal for Exceptional En-gineering Achievement, and two Presidential Rank Awardsfor Meritorious Service one of the highest honors given forgovernment work.

Tim Flores is currently the AssistantManager, Advanced Development Of-fice, Space Launch Systems (SLS) Pro-gram. Prior to SLS he was the Project In-tegration Manager for the Ares V launchvehicle project. He has also served asthe Associate Director of the Indepen-dent Program Assessment Office (IPAO)in the Headquarters Office of ProgramAnalysis and Evaluation (PA&E). Mr.

Flores began his career at NASA in 1990 as a Test Engineerand subsequently held various technical and project man-agement positions. He also worked at L-3 Communicationson a two-year assignment in industry as the Deputy ChiefEngineer and later as the Deputy Project Manager on theNASA SOFIA project. Mr. Flores holds a Bachelors degreein Aerospace Engineering from University of Texas at Ar-lington and a Masters in Engineering and Management fromMassachusetts Institute of Technology (MIT).

5

Jason Hundley provides engineeringsupport to NASA exploration systemsand the Space Launch System program.Since receiving a B.S. in Physics andMathematics from Wright State in 1998and a Masters Degree in Mechanical En-gineering from The George WashingtonUniversity in 2001, he has been involvedin numerous programs in the defense andspace industries. He is a member of the

AIAA Space Transportation Technical Committee, and heis a founder and owner of Zero Point Frontiers Corp. inHuntsville, AL.

Stuart Feldman is a founder and ownerof Zero Point Frontiers Corp, an engi-neering firm in Huntsville, AL specializ-ing in design integration, systems analy-sis, architecture development, and deci-sion management. Mr. Feldman is pro-viding engineering support to NASA ex-ploration systems and heavy lift launchvehicle activities as well as to the NASASpace Launch System program. Mr.

Feldman represented the Launch Vehicle team on the de-velopment of NASA Mars Design Reference Architecture5.0. Mr. Feldman has prior program experience on BigelowSundancer while at Orion Propulsion, Crew Exploration Ve-hicle at Northrop Grumman, and Mars Exploration Rover andC/NOFS at The Aerospace Corporation. Mr. Feldman holds aB.S. in Aerospace Engineering and a Masters of Engineeringin Space Systems, both from the University of Michigan, aswell as an MBA from Pepperdine University, and is currentlypursuing a Ph.D. in Systems Engineering at University ofAlabama in Huntsville.

Timothy S. Monk is a Systems Inte-gration Engineer at the NASA MarshallSpace Flight Center. He is a 2005 grad-uate of Auburn University and a 2006graduate of the University of Coloradoreceiving a Bachelor’s and Master’s De-gree in Aerospace Engineering, respec-tively. He has provided engineering sup-port to NASA since early 2007. Sincethat time, he has performed numerous

trades and analyses on the Ares V, Space Launch System,and other Heavy-Lift Launch Vehicle concepts and potentialmission scenarios, including extensive analysis for the MarsDRA5.0 Launch Vehicle team and co-leading the Ares VMars DRM Assessment team. He is an engineer at Zero PointFrontiers Corporation.

6