Developing input-output simulation metamodels
description
Transcript of Developing input-output simulation metamodels
![Page 1: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/1.jpg)
Developing input-output simulation metamodels
Ken R. McNaught, M.F. Alam and T.J. Ringrose
Cranfield University
Defence Academy
Shrivenham UK
ORS Defence Study Group Presentation
19/10/2005
![Page 2: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/2.jpg)
What is a metamodel?
• A model of a model
• In particular, a simulation metamodel is a simpler auxiliary representation of a simulation model.
• Particularly useful if the simulation model is complex and/or time-consuming to run.
![Page 3: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/3.jpg)
What are metamodels used for?
• Improved understanding
• Speed-up, allowing more cases to be considered, i.e. wider exploration. Trade-off with model accuracy.
• Sensitivity analysis and optimization
![Page 4: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/4.jpg)
Approaches
• Regression models
• Artificial neural networks
• Kriging
• Many others
![Page 5: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/5.jpg)
Regression metamodels• The original metamodelling approach and still the most
widespread.
• See, for example :
Friedman (1996), The Simulation Metamodel, Kluwer, Norwell, Mass.
Kleijnen and van Groenendaal (1992), Simulation – A Statistical Perspective, Wiley, Chichester.
![Page 6: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/6.jpg)
Regression Metamodels II
• If a first order metamodel is assumed, the response variable Y is modelled as
where the are input variables and is the error term.
• Model parameters estimated using least squares. • Procedure supported by many software packages.
1 2 ...Y X X
iX
![Page 7: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/7.jpg)
Regression Metamodels III
• A second order regression metamodel also includes quadratic and two-variable interaction terms:
2
,i i ii i ij i j
i i i j
Y X X X X
![Page 8: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/8.jpg)
Artificial Neural Networks I
• Train ANN to ‘learn’ the input-output relationships of the simulation model.
• Present the ANN with the chosen inputs and resultant simulation outputs from a set of training cases.
• Multi-layer perceptron ANN ‘learns’ by applying back-propagation algorithm until its outputs converge on the known target values.
• Over-fitting avoided by use of a validation set of cases not used directly in the model fitting.
• Training stops when the ANN performance is comparable on both sets of cases.
![Page 9: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/9.jpg)
ANNs II
• Study by Hurrion and Birgil (1999), reported in JORS 50, pp 1018-1033.
• Developed ANN and regression metamodels of two manufacturing systems.
• Demonstrated that selecting training cases by simple random sampling of input parameters can lead to better ANN metamodels than standard factorial designs.
![Page 10: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/10.jpg)
Experimental Design
• Experimental design is an important step in metamodelling.
• Concerned with deciding how the training cases will be selected – what pattern?
• The input variables are called factors and the output variables are called responses.
• The value assigned to a factor during a particular run is called its level.
![Page 11: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/11.jpg)
Some Possible Experimental Designs
• Full factorial
• Fractional factorial
• Central composite
• Simple random sample
• Latin hypercube sample (LHS)
![Page 12: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/12.jpg)
Initial Comparison of Experimental Designs
• Previous work reported in Simulation Modelling Practice and Theory 12, pp 559-578
• Involved comparisons of full factorial, fractional factorial, central composite, random sampling and Latin Hypercube designs
• ANN metamodels developed from a simple deterministic System Dynamics -based combat model
• Latin Hypercube design performed best
![Page 13: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/13.jpg)
Latin Hypercube Sampling
• Suggested by McKay et al (1979) in Technometrics 21, pp 239-245.
• Basic idea is to stratify the range of each factor into the same number of non-overlapping intervals. This number is the same as the number of design points or the sample size, .
• Then a representative value (level) is generated for each interval of each factor by uniform random sampling.
• Finally, each design point is generated by randomly sampling without replacement a level of each factor. This process is repeated until all design points are generated.
sn
sn
![Page 14: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/14.jpg)
A simple random sample
Here we have
2 factors, X
and Y and 5
design points
generated
randomly
from within the
design space.
X210
4
3
2
1
+
+
++
+
Y
![Page 15: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/15.jpg)
A Latin hypercube sample
Here X and Y have each been split into 5 intervals and each design point chosen by randomly pairing intervals. X
Y
0 2
1
2
3
4
0.4 1.61.20.8~U(0,2)
~U(0,4)
LHS: First pairing
+
+
+
++
![Page 16: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/16.jpg)
And another one…
Here we have another possible LHS replicate.
X
Y
0 2
1
2
3
4
0.4 1.61.20.8~U(0,2)
~U(0,4)
LHS: Second pairing
+
+
+
+
+
![Page 17: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/17.jpg)
Screening Designs
The aim of a screening design is to reduce the number of factors included in the main experiment, i.e. to identify and screen out those which are relatively unimportant.
Unimportant in this context means a variable which has little impact on the response variable of interest.
![Page 18: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/18.jpg)
Some Screening Designs
• Cotter’s method
• Sequential bifurcation
• Trocine and Malone’s method
• Morris’ method
![Page 19: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/19.jpg)
Morris’ Method
• See Morris (1991), Technometrics 33, pp 161-174.• Idea behind this method is to identify factors with
a large main effect, factors with a non-linear / interaction effect and those with neither.
• The latter can then be screened out.• It involves estimating means and variances of
response variable differences for each possible factor.
![Page 20: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/20.jpg)
Outline of Morris’ Method
Starting from a randomly chosen basepoint, in the design space, factor i
is selected randomly, increased or decreased by and an elementary effect of that factor calculated as
(1) (1) (1)1( ) [ ( ,..., ) ( )]/i i kd y x x x y (1) (1)x x
(1)x
![Page 21: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/21.jpg)
Let be the new point in the design spacedefined above.
Now, a different factor, j, is selected and a new point defined such
Then:
This continues until an elementary effect has been calculated for each factor.
(2)x
(3)x (3) (2)j jx x
(2) (3) (2)( ) [ ( ) ( )]/jd x y x y x
![Page 22: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/22.jpg)
After an effect has been calculated for each
factor, a new base point is chosen and the
whole process is repeated r times.
Hence, for each factor there are r elementary
effects calculated. Their means and variances
indicate which factors are important.
![Page 23: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/23.jpg)
Outline of Approach Taken in Developing Metamodel of SIMBAT
• Initial pre-screening of factors based on intended use of metamodel.
• Elicitation of bounds for remaining factors based on intended use of metamodel.
• Formal screening of remaining factors using Morris’ method.
• Use of Latin hypercube sampling to construct training set for ANN.
• Development and testing of ANN metamodel.
![Page 24: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/24.jpg)
The Scenario
• The scenario is a meeting engagement between two forces, denoted by Red and Blue, each of approximately battalion size.
• The scenario involves a large number of both direct/indirect fire parameters and human factors.
• Within SIMBAT a unit is the main element of a particular scenario. A unit is composed of a number of components, which are the lowest levels represented in SIMBAT, and the direct fire weapon platforms.
• A component usually represents a vehicle or a grouping of infantry. When a unit has no components left, it is considered ‘dead’ and therefore disappear from the scenario.
![Page 25: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/25.jpg)
Factors Remaining After Pre-Screening
These related to the number of components
within various units on either side, and human
factors such as probability of shock,
probability of surprise and unit participation.
Some of these were grouped, e.g. the
number of components in C and D tank squadrons
(both part of the same battlegroup)
![Page 26: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/26.jpg)
Factor Levels
• For the screening design, we employed 6 levels of each factor.
• For example, the number of tanks in C or D Squadron could be 6, 8, 10, 12, 14 or 16.
• The number of components in the Recce Company could be 6, 10, 14, 18, 22, 26.
• The probability of surprise could be 0, 0.2, 0.4, 0.6, 0.8, or 1.
![Page 27: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/27.jpg)
Results from Morris’ Methodr =1 r=2 r =3 r =4 r =5 r =6 µ σ2
1 0.07 0.03 0.07 0.1 0.15 0.07 0.08 0.002
2 0.58 0.5 0.62 0.22 0.33 0.45 0.45 0.023
3 0.07 0.08 0.10 0.02 0.05 0.13 0.08 0.001
4 0.05 0.08 0.00 0.05 0.12 0.03 0.06 0.002
5 0.05 0.20 0.05 0.10 0.08 0.18 0.11 0.004
6 0.17 0.28 0.05 0.12 0.17 0.08 0.15 0.007
7 0.03 0.02 0.03 0.08 0.02 0.02 0.03 0.001
8 0.28 0.23 0.10 0.30 0.47 0.05 0.24 0.023
9 0.07 0.12 0.08 0.08 0.02 0.02 0.07 0.002
10 0.45 0.18 0.53 0.42 0.43 0.37 0.40 0.014
11 0.15 0.12 0.12 0.05 0.33 0.18 0.16 0.009
12 0.10 0.17 0.17 0.03 0.2 0.05 0.12 0.005
13 0.03 0.32 0.05 0.37 0.2 0.17 0.19 0.019
![Page 28: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/28.jpg)
Identifying the Important Factors
![Page 29: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/29.jpg)
Most Important Factors
• Factor 2: Number of tanks in friendly C or D Squadron
• Factor 8: Number of tanks in OPFOR companies
• Factor 10: Number of components in OPFOR Recce Company
• Factor 13: Unit participation
![Page 30: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/30.jpg)
Middle-Ranking Factors
• Factor 11: Probability of shock
• Factor 6: Size of Anti-Tank Section
• Factor 12: Probability of surprise
• Factor 5: Number of components in friendly Recce Troop
![Page 31: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/31.jpg)
Low-Ranking Factors
These related to the number of components of
dismounted infantry on each side and the
number of components in each side’s HQ.
Clearly, the rank ordering might have been
different in another scenario.
![Page 32: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/32.jpg)
Developing simulation metamodels
• Simulation metamodels were developed using the 8 important factors, derived from Morris’ design
• Main response of interest was Loss Exchange Ratio, LER=Red Casualties/Blue Casualties
• Multi-layer perceptron ANNs considered as potential metamodeling method for the underlying simulation
• A modified Latin Hypercube design (LHD) was employed to generate the required design points to develop candidate metamodels
• 3 different approaches for ANN metamodels were considered
![Page 33: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/33.jpg)
Approaches Employed
• Approach 1: LER used as the target variable for the ANN
• Approach 2: Blue casualties and Red casualties used as the target variables in two separate ANNs
• Approach 3: Blue casualties and Red casualties used as target variables in the same ANN
• LER arrived at indirectly for the 2nd and 3rd approaches.
![Page 34: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/34.jpg)
Approaches (cont’d)
• In each approach, 200 simulation configurations (design points) were used, based on LHS design.
• A further 200 configurations were randomly sampled - 100 used as a validation set and 100 used as an independent test set.
• Each configuration incurred 40 replications.• Separate ANNs were developed for each approach
using the mean results and the full results across all replications (referred to as mean and replication networks).
![Page 35: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/35.jpg)
Performance Measures
●
●
●
1
1 ˆ| | /N
i i ii
MAPD Y Y YN
2
1
1 ˆ( )N
i ii
MSE Y YN
ˆ /i i iRPE Y Y
( 10%) (# : 0.9 1.1) /i iPDRPE within RPE N
![Page 36: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/36.jpg)
Results – Blue CasualtiesApproach Training
DataMSE PDRPE
(within 20%)
PDRPE
(within
10%)
MAPD
2 Mean 15.72 99 88 5.34
2 All 9.53 98 93 4.32
3 Mean 14.32 99 88 5.34
3 All 10.32 99 94 4.20
![Page 37: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/37.jpg)
Results – Red CasualtiesApproach Training
DataMSE PDRPE
(within 20%)
PDRPE
(within
10%)
MAPD
2 Mean 8.05 99 85 5.35
2 All 9.11 97 89 5.85
3 Mean 12.06 96 69 7.41
3 All 9.17 97 86 5.97
![Page 38: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/38.jpg)
LER ResultsApproach Training
DataMSE PDRPE
(within 20%)
PDRPE
(within
10%)
MAPD
1 Mean 0.0052 95 75 7.98
1 All 0.0120 83 54 12.24
2 Mean 0.0072 93 69 8.85
2 All 0.0065 97 73 7.36
3 Mean 0.009 92 63 10.17
3 All 0.0048 96 78 7.17
![Page 39: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/39.jpg)
Conclusions
• Replication networks usually outperform mean networks.
• From earlier work, ANNs employing training data generated from the Latin Hypercube design appear to outperform ANNs employing other designs.
• Separate ANNs not necessary to predict Blue and Red casualties
![Page 40: Developing input-output simulation metamodels](https://reader035.fdocuments.net/reader035/viewer/2022062409/56814bed550346895db8cfbe/html5/thumbnails/40.jpg)
Conclusions• Metamodels have a role to play as simulations become
increasingly complex.• Regression models, ANNs and other approaches can be
used for this purpose – regression is more appropriate if explanation is the goal; ANNs may be more appropriate if prediction is the goal.
• Experimental design is crucial in metamodel development. Different designs may be required for different approaches.
• Factor screening may also be required as part of the overall design process if many potential factors are present in the simulation.