A Calibration and Validation Process A Calibration and Validation Process (CAVP) for Complex Adaptive System (CAVP) for Complex Adaptive System

SimulationSimulation

Lieutenant Colonel Wayne StilwellLieutenant Colonel Wayne StilwellUnited States ArmyUnited States Army7 September 20067 September 2006

Dr. Donald E. Brown, AdvisorDr. Donald E. Brown, AdvisorDr. William T. Scherer, ChairDr. William T. Scherer, Chair

Dr. Stephanie GuerlainDr. Stephanie GuerlainDr. Paul ReynoldsDr. Paul Reynolds

COL (Dr.) George F. Stone, IIICOL (Dr.) George F. Stone, III

Degree Requirements Degree Requirements Course workCourse work completed in May 2005 completed in May 2005 SeminarSeminar in Honolulu, Hawaii 2-10 FEB 2006: in Honolulu, Hawaii 2-10 FEB 2006:

Project Albert 11Project Albert 11thth International Workshop for International Workshop for Agent-Based SimulationAgent-Based Simulation• 13th PAIW in Netherlands 12-17 NOV 200613th PAIW in Netherlands 12-17 NOV 2006

Will lead a team of researchers into command agent Will lead a team of researchers into command agent simulation calibration experimentationsimulation calibration experimentation

ArticleArticle submitted to: submitted to:• Journal of Defense Modeling and SimulationJournal of Defense Modeling and Simulation

““A Calibration and Validation Process (CAVP) for Complex A Calibration and Validation Process (CAVP) for Complex Adaptive System SimulationAdaptive System Simulation

Planned Articles:Planned Articles:• IEEE Journal (Proof adapted from Luenberger 1973)IEEE Journal (Proof adapted from Luenberger 1973)• MORS (Replication of a live experiment with agent-based MORS (Replication of a live experiment with agent-based

simulation)simulation)• ““Command Agent Calibration” - 13Command Agent Calibration” - 13thth PAIW PAIW

CAVPCAVP CAVP is an iterative, information CAVP is an iterative, information

engineering-based process that calibrates engineering-based process that calibrates CASS agent parameters to a range of CASS agent parameters to a range of acceptable outputs.acceptable outputs.

CAVP relies on:CAVP relies on:• Empirical data or Expert opinion on real systemEmpirical data or Expert opinion on real system• Response surface methods (to include ERSM), Response surface methods (to include ERSM),

data mining tools such as classification trees data mining tools such as classification trees and linear regression, NOLH design of and linear regression, NOLH design of experiments, and expert opinion of reasonable experiments, and expert opinion of reasonable inputinput

Problem StatementProblem Statement

Simulation validation techniques do not Simulation validation techniques do not currently offer an ability to:currently offer an ability to:• Measure the influence of non-linear Measure the influence of non-linear

relationships that contribute to the outcome of relationships that contribute to the outcome of a dynamic systema dynamic system

• Reduce the complexity of higher order Reduce the complexity of higher order interactioninteraction

• Calibrate multiple simulation inputs to desired Calibrate multiple simulation inputs to desired outputsoutputs

• Validate a CAS via the entire component Validate a CAS via the entire component comparison before white-box validationcomparison before white-box validation

Complex Adaptive SystemComplex Adaptive System

Agent-basedAgent-based HeterogeneousHeterogeneous DynamicDynamic FeedbackFeedback OrganizationOrganization EmergenceEmergence

•Non-linear interactionNon-linear interaction•Non-reductionismNon-reductionism•Emergent behaviorEmergent behavior•Hierarchical StructureHierarchical Structure•Decentralized ControlDecentralized Control•Self OrganizationSelf Organization•Non-equilibrium OrderNon-equilibrium Order•AdaptationAdaptation•Collectivist DynamicsCollectivist Dynamics

MOUT as a Complex AdaptiveMOUT as a Complex Adaptive SystemSystem

Literature Review Key AuthorsLiterature Review Key Authors

Calibration DefinitionCalibration Definition

The process of adjusting parameter The process of adjusting parameter values in the simulation model to values in the simulation model to better represent the underlying better represent the underlying systemsystem

““Calibration” implies the existence of Calibration” implies the existence of a standard to judge against.a standard to judge against.

Validation Definition (DMSO)Validation Definition (DMSO)

The quality of being inferred, The quality of being inferred, deduced, or calculated correctly deduced, or calculated correctly enough to suit a specific purpose.enough to suit a specific purpose.

The degree of validity is the level of The degree of validity is the level of trust a simulation user can place in trust a simulation user can place in the output of the model.the output of the model.

Classic Validation ConceptClassic Validation Concept

White Box first: Validate each White Box first: Validate each module according to its componentsmodule according to its components

Black Box next: Compare the total Black Box next: Compare the total system output to actual system system output to actual system outputoutput

Literature Review Key PointsLiterature Review Key Points Aggregation based simulations are an Aggregation based simulations are an

improvement over differential equations-based improvement over differential equations-based simulations when modeling complex phenomenasimulations when modeling complex phenomena

CAS require more sophisticated validation CAS require more sophisticated validation methodologies than are currently available to methodologies than are currently available to improve the value of decisionsimprove the value of decisions

Behavioral input of each agent creates emergent Behavioral input of each agent creates emergent behavior, requiring more extensive validation behavior, requiring more extensive validation techniquestechniques

Statistical approaches like the ERSM can provide Statistical approaches like the ERSM can provide the basis for an improved validation method.the basis for an improved validation method.

ERSM (Schamburg and Brown )ERSM (Schamburg and Brown )

NOLHNOLH

The CAVPThe CAVP Determine CAS for investigationDetermine CAS for investigation Examine extant system outputExamine extant system output Determine Measures of Performance (MOP)Determine Measures of Performance (MOP) Develop InputsDevelop Inputs Construct the CASSConstruct the CASS Determine a NOLH DOEDetermine a NOLH DOE Compare MOP using Metrics of Evaluation (MOE)Compare MOP using Metrics of Evaluation (MOE) Conduct Global Convergence Optimization on Conduct Global Convergence Optimization on

responses not in toleranceresponses not in tolerance Declare calibration state; If not calibrated, use Declare calibration state; If not calibrated, use

CART to determine causality of inputsCART to determine causality of inputs

CAVP ProofCAVP Proof

Iterative Composite Mapping Iterative Composite Mapping

The ExperimentThe Experiment

Recreate live soldier firefight CAS (blank rounds with Recreate live soldier firefight CAS (blank rounds with sensors) via a CASSsensors) via a CASS

4 varying scenarios4 varying scenarios 5 Measures of Performance (Blue casualties, red 5 Measures of Performance (Blue casualties, red

casualties, blue rounds fired, red rounds fired, time in casualties, blue rounds fired, red rounds fired, time in seconds)seconds)

Metric of Evaluation (distance function) Metric of Evaluation (distance function) Used MANA as simulation of choiceUsed MANA as simulation of choice NOLH-based DOE, 33 design points, 200 iterations per NOLH-based DOE, 33 design points, 200 iterations per

design point.design point. Heterogenous soldiersHeterogenous soldiers Expert opinion on input ranges for 10 control variablesExpert opinion on input ranges for 10 control variables Exogenous variables held steady throughout Exogenous variables held steady throughout

experimentexperiment

AnalysisAnalysis

Input ParametersInput ParametersVariable Abbreviation Reasonable Range Reason for Range

Blue Speed BLSPEED 10-100 Blue may choose to stalk quietly (slow speed), or move rapidly

Blue Accuracy BLACC 5-60 Blue soldiers are well trained, but may fire less accurately when rapidly acquiring targets in a constrained space.

Blue Detection Range BLUEDET 50-200 Blue soldiers have good sensors and vision, but urban terrain limits detection maximum.

Blue Classification Range BLUECL 30-80 Blue soldiers are well trained, , but may have trouble distinguishing between a non-uniformed enemy and a civilian.

Blue Concealment BLUECONC 15-45 Blue can hide their movements in some respects, but must enter doorways and expose themselves to fire at critical points.

Red Speed REDSPEED 10-60 Red is relatively stationary, but can move quickly within the room if necessary.

Red Accuracy REDACC 5-35 Red soldiers are not as accurate as blue soldiers. However, the close range of the engagements may reduce that disadvantage.

Red Detection Range REDDET 50-200 Red soldiers are inside of buildings and can see out slightly better than blue soldiers can see in.

Red Classification Range REDCL 45-80 Red soldiers can recognize blue soldiers as far as maximum vision allows

Red Concealment REDCONC 40-75 Red can better hide and conceal themselves on the defense and inside of urban terrain.

MOP Target Values MOP Target Values

Response (Ej) Target (μ Ej)=T

σ γ (Lower)=L(μEj-1σ)=L

γ (Upper)=U(μEj+1σ)=U

Time Scenario 1 (E1) 18.00 6.95 11.05 24.95

Blue Casualties (E2) .28 .46 .18* .38*

Red Casualties (E3) .72 .46 .62* .82*

Blue Rounds Fired (E4) 8.09 4.34 3.75 12.45

Red Rounds Fired (E5) 7.37 4.58 3.79 11.95

NOLH Design of ExperimentNOLH Design of Experimentlow level 10 10 5 5 30 45 50 50 15 40

high level 100 60 60 35 80 80 200 200 45 75factor name Blue Speed Red Speed Blue Acc Red Acc Blue Clas Red Clas Blue Det Red Det Blue Conc Red Conc

1 100 15 29 11 74 67 153 120 45 642 92 60 12 16 53 52 163 97 42 553 89 32 55 10 32 66 158 55 24 744 61 54 60 17 77 50 172 59 28 435 94 12 31 12 64 70 111 134 17 446 97 57 22 13 52 53 73 181 16 597 72 33 58 13 30 68 106 186 41 408 58 44 57 15 75 54 83 200 31 739 69 23 17 21 66 56 50 78 33 61

10 78 43 20 26 41 64 64 106 40 4711 75 21 46 34 47 47 69 73 26 6212 80 46 41 33 68 79 120 111 22 4513 63 18 15 22 60 49 195 167 25 5014 86 40 26 31 38 65 191 158 23 6715 66 19 51 32 49 45 148 163 37 5216 83 41 38 35 71 77 134 148 39 6617 55 35 33 20 55 63 125 125 30 5818 10 55 36 29 36 58 97 130 15 5119 18 10 53 24 57 73 88 153 18 6020 21 38 10 30 78 59 92 195 36 4121 49 16 5 23 33 75 78 191 32 7222 16 58 34 28 46 55 139 116 43 7123 13 13 43 27 58 72 177 69 44 5624 38 37 7 28 80 57 144 64 19 7525 52 26 8 25 35 71 167 50 29 4226 41 48 48 19 44 69 200 172 27 5427 33 27 45 14 69 61 186 144 20 6828 35 49 19 6 63 78 181 177 34 5329 30 24 24 7 43 46 130 139 38 7030 47 52 50 18 50 76 55 83 35 6531 24 30 39 9 72 60 59 92 38 4832 44 51 14 8 61 80 102 88 23 6333 27 29 27 5 39 48 116 102 21 49

Sample ResultSample Result

Design Point

Mahala

nobis

Dis

tance

3330272421181512963

25

20

15

10

5

0

VariableGamma DistanceTime Distance

Time Response for Scenario 1

d(Cij, Ej)≤Υj

Results Scenario 1Results Scenario 1

Design Point

Ma

ha

lan

ob

is D

ista

nce

3330272421181512963

25

20

15

10

5

0

VariableGamma DistanceTime Distance

Time Response for Scenario 1

Design Point

Ma

ha

lan

ob

is D

ista

nce

3330272421181512963

200

150

100

50

0

VariableGamma DistanceRed Rounds Distance

Red Rounds Response for Scenario 1

Design Point

Ma

ha

lan

ob

is D

ista

nce

30272421181512963

250

200

150

100

50

0

VariableGamma DistanceBlue Rounds Distance

Blue Rounds Response for Scenario 1

Design Point

Ma

ha

lan

ob

is D

ista

nce

30272421181512963

7

6

5

4

3

2

1

0

VariableGamma DistanceRed Casualty Distance

Red Casualty Response for Scenario 1

Design Point

Ma

ha

lan

ob

is D

ista

nce

30272421181512963

7

6

5

4

3

2

1

0

VariableGamma DistanceBlue Casualty Distance

Blue Casualty Response for Scenario 1

TIME RED RDS BLUE RDS RED CAS BLUE CAS

d(Cij, Ej)≤Υj

CART Analysis of Red Rounds, Scenario 1CART Analysis of Red Rounds, Scenario 1

Variable Score

BLSPEED 100.00 ||||||||||||||||||||||||||||||||||||||||||

BLCONC 32.12 |||||||||||||

REDSPEED 8.39 |||

REDACC 5.27 |

BLACC 2.97

REDCL 2.88

BLCL 1.70

REDDET 0.24

BLDET 0.00

REDCONC 0.00

Scenario # Regression Equation of Significant Factors

P-Value

Scenario 1 time = 260 - 2.01 blue speed

.00

Scenario 2 time = 299 - 4.57 blue speed

.00

Scenario 3 time = 119 - 0.617 blue speed

.00

Scenario 4 time = 69.1 - 0.431 blue speed

.00

Classification Tree Regression Equation

Design Point

Mahala

nobis

Dis

tance

3330272421181512963

200

150

100

50

0

VariableGamma DistanceRed Rounds Distance

Red Rounds Response for Scenario 1

ConclusionsConclusions

MANA will calibrate three of the five MANA will calibrate three of the five MOPsMOPs

MANA is not valid for use unless rate MANA is not valid for use unless rate of fire can be changed in the of fire can be changed in the simulationsimulation

If the simulation can be changed, it If the simulation can be changed, it may come into calibration for the may come into calibration for the final two MOPs, and then could final two MOPs, and then could become a valid CASbecome a valid CAS

ContributionsContributions

A process to calibrate agent input against A process to calibrate agent input against an error tolerance for complex adaptive an error tolerance for complex adaptive system simulationssystem simulations

A simulation validation methodology that A simulation validation methodology that uses a reverse order from classical uses a reverse order from classical validation methodologiesvalidation methodologies

A composite-mapping process that A composite-mapping process that efficiently searches a problem space and efficiently searches a problem space and guides a simulation developer towards guides a simulation developer towards more effective simulationsmore effective simulations

CAV Process ConclusionsCAV Process Conclusions

Relates CASS output back to agent Relates CASS output back to agent inputs and can effectively calibrate a inputs and can effectively calibrate a simulationsimulation

Determines the calibration state of Determines the calibration state of agents, and determines the agents, and determines the validation state of the CASvalidation state of the CAS

Can guide the modeler and inform Can guide the modeler and inform the simulation development processthe simulation development process