Dimensions of Credibility in Models and Simulationsaegistg.com/Technical_Papers/08E-SIW-076...
Transcript of Dimensions of Credibility in Models and Simulationsaegistg.com/Technical_Papers/08E-SIW-076...
ЄDINBVRGH
Dimensions of Credibility in Models and Simulations
Martin J. Steele, Ph.D.National Aeronautics and Space Administration (NASA)
IT-C1 / Kennedy Space Center, FL [email protected]
June 2008ЄDINBVRGH
08E-SIW-0762
ЄDINBVRGHOverview
• Recent Historical Perspective• The Idea of Credibility in M&S• Dimensions of Credibility• Concerns• Findings• Summary• Final Thoughts
08E-SIW-0763
ЄDINBVRGHM&S Standard Development
2 Credibility Scales 1 New Credibility Scale
InterimNov 06
Final SubmittedNov 07
CAIBReport
NASAOCE
Direction M&SLiterature
DiazReport
MgtDecision Maker
Interviews
ExternalEfforts
NASA-wideFormal Review
PilotStudies
08E-SIW-0764
ЄDINBVRGHRecent Historical Perspective
Inside NASA• The NASA Standards
for Models & Simulations – 2007 Summer Computer Simulation Conference– Interim Version
• Development of NASA’s Models and Simulations Standard – 2008 Spring Simulation Interoperability Workshop– Final Version
Outside NASA• DoD
– M&S Validation (VPMM)– M&S Quality
• DoE – Predictive Capability Maturity Model (PCMM)
• ASME – M&S V&V in Computational Solid Dynamics
• AIAA – M&S V&V in Computational Fluid Dynamics
08E-SIW-0765
ЄDINBVRGHGoals for Excellence in M&S
Goals of the M&S Standard• Ensure that the credibility of M&S results
is properly conveyed to those making critical decisions
• Assure that the credibility of M&S meetsthe project requirements
Additionally,• Form a strong foundation for disciplined
development, validation and use of M&S
• Include a standard method to assess the credibility of the M&S presented to the decision maker
ЄDINBVRGH
The Idea of Credibility in M&S
08E-SIW-0767
ЄDINBVRGH
Accreditation
Verification Validation
Development© Sargent, R. G. (c. 1980).© Sargent, R. G. (c. 1980).
Development Foundation
08E-SIW-0768
ЄDINBVRGHOperational Foundation
Obtaining Credible Results Requires:
• ‘Good’ Input• Understanding of
Uncertainty• Sensitivity Analysis
Ver Val
Development
InputPedigree
UncertaintyQuant. Robustness
Operations
Accreditation
Results
08E-SIW-0769
ЄDINBVRGHSupporting Evidence
Adding to Credibility:• Past Use• M&S Mgt• Developer / User /
Analyst Qualifications
Ver Val
Development
InputPedigree
UncertaintyQuant. Robustness
Operations
ResultsAccreditation
UseHistory
ModelMgt
PeopleQual.
Supporting Evidence
08E-SIW-07610
ЄDINBVRGHBreadth to Consider
• Problem Statement• System Understanding• Modeling Choice• Conceptual Model
Accreditation
Ver Val InputPedigree
UncertaintyQuant. Robustness Use
HistoryModelMgt
PeopleQual.
Development Operations Supporting Evidence
Results
System Understanding
Real World System
Computational Model
Conceptual Model
SystemAnalysis
Verification
ValidationValidation
ImplementableConceptual Model
Coding theModel Verification
Validation
DetailedUnderstanding
ЄDINBVRGH
08E-SIW-07612
ЄDINBVRGHVerification & Validation
Verification• Structure• Flow• Fidelity
• How:– Comparing to
Conceptual Model– Code Tracing– Primitive Tests– Min/Max Value Tests
Validation: “… determining the degree to which a model or a simulation is an accurate representation of the real world …”
08E-SIW-07613
ЄDINBVRGHInput Pedigree
Input:• Source
– Notional– Subject Matter Expert– Applicability to current
problem• Referent Quality
relative to current problem
– Referent System– Referent Environment
• Quantity
• Authoritative Data
Input Form:• What’s the character of
your analysis?
– Average
– Uniform
– Triangular
– Curvilinear
08E-SIW-07614
ЄDINBVRGHAccuracy & Uncertainty
Accuracy:
Uncertainty:• Sources• ‘Size’• How Confident
• Epistemic– Reducible– Subjective– Model Form– Incomplete Information
• Aleatory– Irreducible– Variability– Inherent – Stochastic
True Value ‘Modeled’ Value
Uncertainty in‘Modeled’ Value
Uncertainty inTrue Value
08E-SIW-07615
ЄDINBVRGHRobustness
Robustness of Results, i.e.,Sensitivity of:
• The Real World System (RWS)
• The M&S
Not a Good Situation
M&S is not robust, butRWS is
- Validation Issue- Results will be overly
conservative
Best Situation
RWS is robust (Insensitiveto Changes)
&the M&S matches
the RWS
Worst Situation
M&S shows aRobustness not present
in the RWS- Validation Issue
- M&S not so useful
OK Situation
RWS is sensitive to change& the M&S matches
the RWS
RWSM
&S
InsensitiveTo Changes
SensitiveTo Changes
SensitiveTo Changes
InsensitiveTo Changes
Better
08E-SIW-07616
ЄDINBVRGHUse History & Management
Use History:• Similarity of Uses
– Analogous Systems– Exact Systems
• Length of Time in Use– Just Developed
• Just Updated– Long-Term Successful
Use
M&S Management:• Models & Data under
Configuration Control• Models are
– Maintained– Sustained
08E-SIW-07617
ЄDINBVRGHPeople Qualifications & Tech Review
People Qualifications:• Education• Training• Experience
• In M&S• With the Modeled (Real
World) System
Technical Review:• When accomplished
– During M&S Development– During M&S Operations
• Qualifications & Independence of the ‘Peer’Review Group:– Self– Internal Organization– External– Non-Expert to Expert
• Level of Formalism– Planning– Documentation
Ver Val
Development
InputPedigree
UncertaintyQuant. Robustness
Operations
InputPedigree
UncertaintyQuant. Robustness
Operations
ЄDINBVRGH
When to Apply the Standard?
08E-SIW-07619
ЄDINBVRGHScope of the M&S Standard
• Standard covers the use of M&S affecting:• Human Safety• Mission Success
• The focus is on M&S for flight and ground support projects– Operations– Test & Evaluation– Manufacturing and Assembly– Design and Analysis
Critical Decisions
As defined by each Program
4: Controlling 3: Significant 2: Moderate 1: Minor
M&S Results Influence
0: None IV: Negligible III: Marginal II: Critical I: Catastrophic
Project Consequence
Sample Risk Matrix
08E-SIW-07620
ЄDINBVRGHBasic Ideas
• Documentation of M&S Activities (Sections 4.1 – 4.6)
• Credibility Assessment (Section 4.7 & Appendix A)
• Reporting to Decision Makers (Section 4.8)– M&S Analysis Results– A statement on the uncertainty in the results– Credibility of M&S Results– Identify
• Unfavorable outcomes• Violation of assumptions
– Unfavorable Use Assessment• Difference Between V&V & Use Assessment
08E-SIW-07621
ЄDINBVRGHUse Assessment
Expected OutputRange
Intended Input Domain
ValidatedInput Domain
ValidatedOutput Range
Note – this is a 2-dimensional example of a potentially multi-dimensional input domain & multi-dimensional output range
ЄDINBVRGH
Sommaire& Future Directions
08E-SIW-07623
ЄDINBVRGHFuture Directions
• Track 1 – Internal – Deploy– Use of Standard /
Collection of Data on Use– Assess– Revise
• Track 2 – M&S Discipline Specific
– Develop Guides (RPGs, Handbooks)
– For each M&S type in relation to the NASA M&S Standard
• Track 3 – External Collaboration
– Other M&S Standards/Guides
– Professional/Academic Organizations
ЄDINBVRGH
Broadening the Perspective of M&S V&V
and Simplifying It !
08E-SIW-07625
ЄDINBVRGHNetwork Layered Protocol Approach
Like the Layered Network Protocol
Model
08E-SIW-07626
ЄDINBVRGHLayered M&S View(Influences in M&S Results)
Computer Hardware
Operating System Software
Application Software
Model / Simulation
M/S Input M/S OutputM&S V&Vand
Credibility Assessment
Analyzing Outputincluding Post-Processing of Output Data
Industry Standardsand
Broad Use
User Inputincluding
Run Setup
Need for a Clearinghouse for Commercial & Open Source M&S Languages & Application Software
ЄDINBVRGH
Final Thoughts
08E-SIW-07628
ЄDINBVRGHConcerns & Findings
• General Concerns– Broad Applicability– Subjectivity– Software Similarity– The Analysis vs. The
People
• If we don’t have this Standard– Many varieties of
Credibility Assessment will be are being presented
• Findings– Yet Another Standard– Terminology
(Denotations & Connotations
• Types• Purposes• Education, Experience,
& Training
08E-SIW-07629
ЄDINBVRGHWhy doesn’t/can’t the Software Engineering Req’ts do the job?
• Models and simulations may be implemented in software, butthey encompass much more than software engineering
• The SW Engineering NPR (7150.2) does not address many issues critical for M&S, e.g.,– development of models– validation against experimental or flight data– uncertainty quantification– operations and maintenance of M&S
• Models are sometimes developed in hardware– Physical Models (Scaled)– Analog Models
SoftwareEngineering
Models &Simulations
08E-SIW-07630
ЄDINBVRGHComparison
Software• Performs a task within a
system
• Purpose – performance of tasks/functions for a system
• Requirements are functionsthe s/w shall perform– More discrete functions in the
code (function-centric)• Software shall do this• S/W shall do that• Etc.
• Traditional s/w functions either work or don’t work– Typically, there is no
uncertainty
• Totally focused on Software Development
M&S• Provides a representation of a
system
• Purpose – Analysis orRepresentation of a system or of system functions for insight
• Requirements are behaviorsthe simulation model shall exhibit– System behavior-centric
• Includes more ‘assumptions’how different parts of the simulation model work– Typically includes
uncertainty in the behavior of the simulation
• Must include requirements (& reporting) on M&S Use
08E-SIW-07631
ЄDINBVRGHSoftware & M&S
• You can follow software standards correctly in model development– BUT STILL produce
inaccurate or miss-leading results
• Software Standards do not deal with Software (or Model) use !
08E-SIW-07632
ЄDINBVRGHFinal Statements
• Communicating Results• Framework for Discussion of Results Credibility• Credibility Factors Beyond VV&A
– With dimensions composing each factor of M&S Credibility
• It is:– High Level– Broadly Applicable (To All M&S Types!)
Accreditation
Ver Val InputPedigree
UncertaintyQuant. Robustness Use
HistoryModelMgt
PeopleQual.
Development Operations Supporting Evidence
Results
Technical Review
ЄDINBVRGH
Thank You …
ЄDINBVRGH