A Calibration and Validation Process (CAVP) for Complex Adaptive System Simulation
Embed Size (px)
Transcript of A Calibration and Validation Process (CAVP) for Complex Adaptive System Simulation
A Calibration and Validation Process (CAVP) for Complex Adaptive System SimulationLieutenant Colonel Wayne StilwellUnited States Army7 September 2006
Dr. Donald E. Brown, AdvisorDr. William T. Scherer, ChairDr. Stephanie GuerlainDr. Paul ReynoldsCOL (Dr.) George F. Stone, III
Degree Requirements Course work completed in May 2005Seminar in Honolulu, Hawaii 2-10 FEB 2006: Project Albert 11th International Workshop for Agent-Based Simulation13th PAIW in Netherlands 12-17 NOV 2006Will lead a team of researchers into command agent simulation calibration experimentationArticle submitted to:Journal of Defense Modeling and SimulationA Calibration and Validation Process (CAVP) for Complex Adaptive System SimulationPlanned Articles:IEEE Journal (Proof adapted from Luenberger 1973)MORS (Replication of a live experiment with agent-based simulation)Command Agent Calibration - 13th PAIW
CAVPCAVP is an iterative, information engineering-based process that calibrates CASS agent parameters to a range of acceptable outputs.
CAVP relies on:Empirical data or Expert opinion on real systemResponse surface methods (to include ERSM), data mining tools such as classification trees and linear regression, NOLH design of experiments, and expert opinion of reasonable input
Problem StatementSimulation validation techniques do not currently offer an ability to:Measure the influence of non-linear relationships that contribute to the outcome of a dynamic systemReduce the complexity of higher order interactionCalibrate multiple simulation inputs to desired outputsValidate a CAS via the entire component comparison before white-box validation
Complex Adaptive SystemAgent-basedHeterogeneousDynamicFeedbackOrganizationEmergence
Non-linear interactionNon-reductionismEmergent behaviorHierarchical StructureDecentralized ControlSelf OrganizationNon-equilibrium OrderAdaptationCollectivist DynamicsMOUT as a Complex Adaptive System
Literature Review Key Authors
Calibration DefinitionThe process of adjusting parameter values in the simulation model to better represent the underlying systemCalibration implies the existence of a standard to judge against.
Validation Definition (DMSO)The quality of being inferred, deduced, or calculated correctly enough to suit a specific purpose.The degree of validity is the level of trust a simulation user can place in the output of the model.
Classic Validation ConceptWhite Box first: Validate each module according to its componentsBlack Box next: Compare the total system output to actual system output
Literature Review Key PointsAggregation based simulations are an improvement over differential equations-based simulations when modeling complex phenomenaCAS require more sophisticated validation methodologies than are currently available to improve the value of decisionsBehavioral input of each agent creates emergent behavior, requiring more extensive validation techniquesStatistical approaches like the ERSM can provide the basis for an improved validation method.
ERSM (Schamburg and Brown )
The CAVPDetermine CAS for investigationExamine extant system outputDetermine Measures of Performance (MOP)Develop InputsConstruct the CASSDetermine a NOLH DOECompare MOP using Metrics of Evaluation (MOE)Conduct Global Convergence Optimization on responses not in toleranceDeclare calibration state; If not calibrated, use CART to determine causality of inputs
Iterative Composite Mapping
The ExperimentRecreate live soldier firefight CAS (blank rounds with sensors) via a CASS4 varying scenarios5 Measures of Performance (Blue casualties, red casualties, blue rounds fired, red rounds fired, time in seconds)Metric of Evaluation (distance function) Used MANA as simulation of choiceNOLH-based DOE, 33 design points, 200 iterations per design point.Heterogenous soldiersExpert opinion on input ranges for 10 control variablesExogenous variables held steady throughout experiment
MOP Target Values
NOLH Design of Experiment
Sample Resultd(Cij, Ej)j
Results Scenario 1TIMERED RDSBLUE RDSRED CASBLUE CASd(Cij, Ej)j
CART Analysis of Red Rounds, Scenario 1
Classification TreeRegression Equation
ConclusionsMANA will calibrate three of the five MOPsMANA is not valid for use unless rate of fire can be changed in the simulationIf the simulation can be changed, it may come into calibration for the final two MOPs, and then could become a valid CAS
ContributionsA process to calibrate agent input against an error tolerance for complex adaptive system simulationsA simulation validation methodology that uses a reverse order from classical validation methodologiesA composite-mapping process that efficiently searches a problem space and guides a simulation developer towards more effective simulations
CAV Process ConclusionsRelates CASS output back to agent inputs and can effectively calibrate a simulationDetermines the calibration state of agents, and determines the validation state of the CASCan guide the modeler and inform the simulation development process