ABSVal
description
Transcript of ABSVal
1
ABSVal
Goal: Adapt accepted Best Practices of VV&A* to simulation models that...
a. Display emergent behavior
b. Are used to model military effects on population dynamics and social phenomena as well as military decision making
c. Support analyses*maybe refine the general Best Practices
2
B.L.U.F.WHAT I THINK WE LEARNED or CONFIRMED
• VV&A universally reviled.– EXCEPTION: DMSO employees and alumni
• VV&A general principles do not map well to analysis domain.– describing anticipated use is problematic– analysis = exploration
• ABS translates to Models exhibiting Emergent Behavior.– pre-production experimentation focused on achieving top-down control (building predictive
capability) on dynamics that initially display emergence– scientifically examining emergence is interesting
• We can help analysts and decision makers distinguish good analysis from bad analysis.
• Minimally acceptable sim-to-app validation is recognizable, achievable, useful, and rare.
3
OUTLINE1. SIMULATION
MODELS
2. A LITTLE EMERGENT BEHAVIOR
3. THINKING ANALYSIS
4. EXPOSING THE MATCH
4
“In contrast to this interest in model-related technology, therehas been far too little interest in the substance of the models and the validity of the lessons learned from using them. In ourview, the DoD does not appreciate that in many cases themodels are built on a base of sand.”
The Base of Sand Problem: A White Paper on the State of Military Combat Modeling.Paul K. DavisDonald Blumenthal
5
WORKABLE DEFINITIONS
• Conceptual Model – Description of the system in some abstracted/symbolic formalism, usually mathematics.
• Verification – The simulation executable faithfully reflects the Conceptual Model
• Validation – The degree to which the system described in the conceptual model is appropriate in supporting the intended use.
• Accreditation – The judgement, made by someone responsible for the outcome, that the simulation is adequately verified and valid for the intended use.
6
COMMENTS
• Conceptual models are always incomplete.
• Verification of a simulation is a scientific endeavor if the conceptual model is complete.
• A simulation is never “Valid.”
• Analytical intended uses are difficult to deal with…– Repetition is very rare.– Analysts have no way to scientifically express the intended use.– Analysts often accept very poor data and models, and often
express grave caveats for their results.
7
formal transitionsT(M)
implementationand design
executablecode
conceptualmodel
ideal sim
naturalsystem
IDEALIZEDDEVELOPMENTPROCESS
8
formal transitionsT(M)
implementationand design
executablecode
conceptualmodel
ideal sim
naturalsystem
IDEALIZEDDEVELOPMENTPROCESS
abstraction
modeling
mapping tosim design pattern
coding andtesting
softwaredesign
9
formal transitionsT(M)
implementationand design
executablecode
conceptualmodel
ideal sim
naturalsystem
IDEALIZEDDEVELOPMENTPROCESS
abstraction
modeling
mapping tosim design pattern
coding andtesting
softwaredesign
Driven by analytictask. More later...
10
executablecode
ideal sim
naturalsystem
REALITY FORBIG-IRON SIMS
abstraction
data development
11
formal transitionsT(M)
implementationand design
executablecode
conceptualmodel
ideal simFOR A GIVEN ANALYSIS
naturalsystem
FOR ANOTHERDAY
FO
CU
S
12
"The more complex the model, the harder it is to distinguish unusual
emergent behavior from programming bugs."
Douglas Samuelson
Renowned Operations
Research Analyst
13
ABSVal PROJECT
• ABSVal Framework
• Test Examples– Pythagoras COIN– SZ/BZ Obstacle Reduction Analysis
• Conclusions
14
VALIDATION
• First-Principles Validation– Assess the theory behind
the conceptual model• And predict its impact on
the ensuing analysis– Examine the
implementation of the theory
• And predict its impact on the ensuing analysis
– Examine the combinations of theories used together
• Results Validation– Compare output data to
data from another source• Historical case• Another model• Intuition
15
VALIDATION
• First-Principles Validation– Assess the theory behind
the conceptual model• And predict its impact on
the ensuing analysis– Examine the
implementation of the theory
• And predict its impact on the ensuing analysis
– Examine the combinations of theories used together
• Results Validation– Compare output data to
data from another source• Historical case• Another model• Intuition
VALIDATION = EXPOSITION + ASSESSMENTThe EVIDENCE on which acceptance is based.
16
formal transitionsT(M)
implementationand design
executablecode
conceptualmodel
ideal sim
naturalsystem•Simulations displaying
emergent behavior are difficult to validate because it is difficult to predict their behavior from the Conceptual Model
•Therefore there is greater pressure to use results validation.
17
“All models are wrong, some are useful.”
George Box
Wartime Statistician
18
ANALYSIS
• Predict the response (absolute)• Predict the response (as compared to a baseline)• Predict the functional form of the response for a set of
independent variables• Predict the sign of the gradient (set of 1st derivatives)• Is there any response?• Predict the min/max of the response over a high-
dimensional domain• Predict xi in [Li, Ui] such that response > c• Characterize the probabilistic nature of the response
Compare to physical sciences -- These are very humble goals.Might a medical/biological mindset be more appropriate?
19
…more ANALYSIS
Provide the best decision support possible, and include useful assessment of the value of the analysis vis. the questions & issues at hand.
20
IDEAL STUDY PROCESS
DETERMINETHE QUESTION
DETERMINETHE QUESTION
DETERMINETHE MOE’s and the EEA’s
DETERMINETHE MOE’s and the EEA’s
FIND or BUILDthe BEST (SIMULATION)MODEL
FIND or BUILDthe BEST (SIMULATION)MODEL
PRODUCTIONRUNS
PRODUCTIONRUNS
PRESENT (and DEFEND)RESULTS
PRESENT (and DEFEND)RESULTS
21
SIMULATION-SUPPORTED ANALYSIS
• Baseline/Excursion or Factorial Experiment
• Driven to answer Analysis Questions
• Key Elements of Analysis
• Constraints, Limitations, and Assumptions
22
Schism• Agent-based simulations use modular rules
and local reasoning to produce realistic and/or interesting emergent aggregate behavior.– Surprise is good**
• Successful simulation testing (core to face/results validation) based on demonstrating credibility across the range of potential input.– Surprise not good**
** Refined later in this talk
23
GOAL: STOP BEING SURPRIZED
Surprise
Explore
Explain
Accept/reject
ProductionRuns
How do w
e te
ll ab
out this
exper
ience
?
In control,no more surprises
1. “Unnatural acts” reflect negatively on a sim
2. Once we achieve top-down control, is there still emergent behavior?
24
ELEMENTSAdaptation of the Yost Scale
SIMULATION DYNAMICS• based on accepted physical
laws• based on accepted social
dynamics• based on common sense• distillation
– simple model relic required to facilitate actions
– simple model relic required to maintain consistency
• top-down human intervention
DATA• authoritative value• measured• witnessed• argued by logic• sensible range• guess/arbitrary• dimensionless
RELEVANT DYNAMICS + REQUIRED DATA = ELEMENTe.g. underwater detection
using observed detection range data in a cookie-cutter model
25
ELEMENTSAdaptation of the Yost Scale
SIMULATION DYNAMICS• based on accepted physical
laws• based on accepted social
dynamics• based on common sense• distillation
– simple model relic required to facilitate actions
– simple model relic required to maintain consistency
• top-down human intervention
DATA• authoritative value• measured• witnessed• argued by logic• sensible range• guess/arbitrary• dimensionless
RELEVANT DYNAMICS + REQUIRED DATA = ELEMENTe.g. underwater detection
using observed detection range data in a cookie-cutter model
CO
NT
RO
LA
BL
E A
BS
TR
AC
TIO
NA
NA
LY
TIC
AL
LY
DE
SIR
AB
LE
26
“It’s the Data, Stupid.”Phalanx, DEC 07George Akst
27
Constraints, Limitations, and Assumptions Guide
TRADOC Analysis Center255 Sedgwick Avenue
Fort Leavenworth, KS 66027-2345
TRAC-TD-05-011 (rev. 1)January 2008
Mike Bauman
28
PARSING SOURCES OF VARIABILITY
CORE: drive the results of your experiment, align with the keyelements of analysis
DYNAMIC CONTEXT: has impact on the circumstances relevant to exercising the core model dynamics, create situations, not elements of analysis
CASES: details necessary tosupport the model, cases to be consideredto achieve analytical goals
C.L.A.: Constraints, Limitations, and Assumptionsnecessary to give scope the analysis,and interpret the results
29
IMPACT ON ANALYSIS
• Agent-based design is reputed to enable fast and easy construction of dynamic context
• Dynamic Context elements can display emergent behavior to add variability
– Emergent behavior is often not predictable/controllable• Big-iron simulations often have parametric (knob) control
over Case elements– impossible to promote these to Dynamic Context or Core
elements– should NOT be elements of analysis
• Ideally, analysts should have the most faith in their Core elements
– should have high-quality data (high on the YOST scale)– should have well-studied dynamics (high on the YOST scale)– must not display uncontrolled emergent behavior
• Limitations on the Core = Limitation of the simulation for analytical purposes
• Core and Dynamic Context Elements should results-proven to be consistent with SME (explainable 1st derivative)
• Core elements should be results-proven to be highly influential (see Scientific Method of Choosing Model Fidelity)
30
IMPACT ON ANALYSIS• Agent-based design is reputed to enable fast and easy construction
of dynamic context• Dynamic Context elements can display emergent behavior to add
variability– Emergent behavior is often not predictable/controllable
• Big-iron simulations often have parametric (knob) control over Case elements
– impossible to promote these to Dynamic Context or Core elements– should NOT be elements of analysis
• Ideally, analysts should have the most faith in their Core elements– should have high-quality data (high on the YOST scale)– should have well-studied dynamics (high on the YOST scale)– must not display uncontrolled emergent behavior
• Limitations on the Core = Limitation of the simulation for analytical purposes
• Core and Dynamic Context Elements should results-proven to be consistent with SME (explainable 1st derivative)
• Core elements should be results-proven to be highly influential (see Scientific Method of Choosing Model Fidelity)
Taxonomy for sources of variability reflecting the relationship between modeldynamics and analytical goals.
** Jargon for communicating how a sim element relates to the analysis.
** Identifies appropriate role for elements withemergent behavior in an analysis.
31
GOLDEN GATE BRIDGE
a solid connection…
simulation/data
capabilities
analytical
requirements
32
TACOMA NARROWS BRIDGE
or, not so much.
simulation/data
capabilities
analytical
requirements
33
Core Experimentaln Parametric Settings
Dynamic
ContextStochastic
1 Parametric Setting
Cases Discrete Cases m Cases
CLA Static1 fixed set of assumptions
RECOMMENDED HANDLING
34
EXAMPLE
• Question: What is the tactical value of LW components to a rifle squad?
• Core: weapon, computer/comm/SA, sight/NVG
• Dynamic Context: paths of maneuver, acquisitions and detections, paths & actions of threat, attrition, …
• Cases: terrain type (urban, jungle, alpine), scale (company, platoon), mission (HVT, defend a FOB)
• CLA: kinetic outcome, unambiguous threat, terrain representation
35
EXAMPLE
• Question: What is the tactical value of LW components to a rifle squad?
• Core: weapon, computer/comm/SA, sight/NVG
• Dynamic Context: paths of maneuver, acquisitions and detections, paths & actions of threat, attrition, …
• Cases: terrain type (urban, jungle, alpine), scale (company, platoon), mission (HVT, defend a FOB)
• CLA: kinetic outcome, unambiguous threat, terrain representation
emergent behavior
dynamics fit here
don’t average over
these cases
36
“Those claims to knowledge that are potentially falsifiable can then be admitted to the body of empirical science, and then further differentiated according to whether they are (so far) retained or indeed are actually falsified.”
Carl PopperPhilosopher of Science
37
NEGATIVE INFORMATION for IN-VALIDATION• Elements not data-driven• Elements not controllable• Element displays undesired emergent
behavior• Element displays unexplainable 1st-order
influence (results schism unexplainable)• Element not in the anticipated layer
– level of influence is more/less than anticipated by analyst
– dynamics or data are...• too low on the Yost scale• mismatched vis. the Yost scale
38
NEGATIVE INFORMATION = IN-VALIDATION?
no concern show stopper
Negative information: scopes analytical value of results.Analyst’s art: responsibly expand this scope.
“This approach uses a very unrealistic model of certain dynamics,but it creates adequate dynamic context to stimulate the core elementsin a way useful to our analytic goals.”
39
“Computer programs should be verified,Models should be validated,
and Analysts should be accredited.”
Alfred G. BrandsteinRenowned Military Operations Research Analyst
Founder of Project Albert
40
THE ANALYST• Prior to any experience with the simulation, can the Analyst...
– Pose analytic questions mathematically?– Describe the experiment?– Identify Core vs. Dynamic Context elements?– Specify CLA elements?– Evaluate Core elements on Yost scale?– Disclose all outcomes the analyst anticipates matching with the simulation (Test
Cases)?• Once experience has been gained, can the Analyst...
– Explain changes to anticipated Core/Dynamic Context/Case/CLA classification?– Describe all testing and tuning required?– Quantify level of influence of each Core & Dynamic Context statistically?– Avoid integrating (averaging) Cases?– Explain the impact of each CLA on the results?– Statistically determine the level of agreement of the simulation outcomes with the
Test Cases?• Resulting analysis should be peer-reviewed
41
Bottom line
42
• Have lots of computational experience with your model.
• Understand and be able to control it emergent behavior.
• Plan and execute experiments, document.
• Disclose relationship between each important sim element and the analytical goal.– Core– Dynamic Context– Cases– CLA
43
QUESTIONS?