The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07...
-
date post
21-Dec-2015 -
Category
Documents
-
view
217 -
download
0
Transcript of The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07...
![Page 1: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/1.jpg)
The AutoSimOA Project
Katy Hoad, Stewart Robinson, Ruth DaviesWarwick Business School
OR49 Sept 07
A 3 year, EPSRC funded project in collaboration with SIMUL8 Corporation.
![Page 2: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/2.jpg)
OUTLINE
IntroductionMethodsAlgorithmTest MethodologyTest ResultsExtended Algorithm & ResultsDiscussionSummary
![Page 3: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/3.jpg)
Objective
To provide an easy to use method, that can be incorporated into existing simulation software, that enables practitioners to
obtain results of a specified accuracy from their discrete event simulation model.
(Only looking at analysis of a single scenario)
![Page 4: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/4.jpg)
Introduction• Initial Setup:
Any warm-up problems already dealt with. Run length (m) decided upon. Modeller decided to use multiple replications to
obtain better estimate of mean performance.
• Multiple replications performed by changing the random number streams used by the model and re-running the simulation.
N
jj
NNm
NN
m
m
XN
X
X
X
XXX
XXX
XXX
1
1
21
222
21
112
11
1
ˆ
ˆ
,,,
,,,
,,,
Output data from modelResponse measure of interest
= summary statistic from rep1
= summary statistic from repN
N replications
![Page 5: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/5.jpg)
QUESTION IS…How many replications are
needed? • Limiting factors: computing time and
expense.
If performing N replications achieves a sufficient estimate of mean performance:> N replications: Unnecessary use of computer
time and money.< N replications: Inaccurate results → incorrect
decisions.
![Page 6: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/6.jpg)
4 main methods found in the literature for choosing N:
1. Rule of Thumb
Run at least 3 to 5 replications.
Advantage: Very simple.
Disadvantage: Does not use characteristics of model output.
No measured precision level.
![Page 7: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/7.jpg)
2. Simple Graphical Method • Plot Cumulative mean -v- number of replications
• Visually select point where cumulative mean line becomes “flat”. Use this as N.
Cumulative mean graph
45
47
49
51
53
55
1 9 17 25 33 41 49 57 65 73 81 89 97 105
Number of replications (n)
Cum
ula
tive m
ean
Advantages: Simple Uses output of interest in decision.
Disadvantages: Subjective No measured precision level.
![Page 8: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/8.jpg)
3. Confidence Interval (with Specified Precision) Method
• User decides size of error they can tolerate.• Run increasing numbers of replications, • Construct Confidence Intervals around sequential
cumulative mean of output variable until desired precision achieved.
Advantages: Relies upon statistical inference to determine
number of replications required.
Allows the user to tailor accuracy of output results to their particular requirement
or purpose for that model and result.
Disadvantage: Many simulation users do not have the skills to apply such an approach.
![Page 9: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/9.jpg)
4. Prediction Formula Method
• User decides size of error they can tolerate.• Run a few replications, estimate variance & mean• Use formula to predict N.
• Check desired precision achieved – if not amend N and
repeat Advantages: Simple.
Uses data from model. Provides specified precision.
Disadvantage: Can be very inaccurate especially for small number of replications.
If variance estimate low underestimate NIf variance estimate high overestimate N
st
NN 1,2
![Page 10: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/10.jpg)
• Chose to automate:
Confidence Interval (with Specified Precision) Method
![Page 11: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/11.jpg)
Run
Model START:
Load Input
Produce Output Results
Run Replication Algorithm
Precision criteria met?
Recommend replication number
Run one more
replication
YES
NO
The replication algorithm interacts with the simulation model sequentially.
![Page 12: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/12.jpg)
2,1 nt
n
nn
nX
nt
d
s2,1
100
is the student t value for n-1 df and a significance of 1-α,
nX
sn is the estimate of the standard deviation,
calculated using results Xi (i = 1 to n) of the n current replications.
Where
n is the current number of replications carried out,
We define the precision, dn, as the ½ width of the Confidence Interval expressed as a percentage of the cumulative mean:
is the cumulative mean,
ALGORITHM DEFINITIONS
![Page 13: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/13.jpg)
Stopping Criteria
• Simplest method:
Stop when dn 1st found to be ≤ desired precision, drequired , and recommend that number of replications, Nsol, to the user.
• Problem: Data series could prematurely converge, by chance, to incorrect estimate of the mean, with precision drequired , then diverge again.
• ‘Look-ahead’ procedure: When dn 1st found to be ≤ drequired, algorithm performs set number of extra replications, to check that precision remains ≤ drequired.
![Page 14: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/14.jpg)
0
20
40
60
80
100
120
140
3 100
137
174
211
248
285
322
359
396
433
470
replication number (n )
f(kLim
it)
kLimit=0 kLimit=5
kLimit=10 kLimit=25
‘Look-ahead’ procedurekLimit = ‘look ahead’ value. Actual number of replications checked ahead is a function of this user defined value:
Function relates ‘look ahead’ period length with current value of n.
100,100
100,)(
nkLimitn
nkLimitkLimitf
![Page 15: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/15.jpg)
23
25
27
29
31
33
35
37
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Replication number (n)
NsolNsol + f(kLimit)
f(kLimit)
Precision ≤ 5%X
X
95% confidence limits
Cumulative mean,
Replication Algorithm
![Page 16: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/16.jpg)
0.8
0.85
0.9
0.95
1
1.05
1.1
1.15
1.2
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
Replication number (n)
Precision
≤ 5%
Precision
> 5%
Precision ≤ 5%
f(kLimit)
NsolNsol + f(kLimit)Nsol
![Page 17: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/17.jpg)
• 24 artificial data sets created: Left skewed, symmetric, right skewed; Varying values of relative standard deviation (stdev/mean).
• Advantage: true mean and variance known.
• Artificial data set: 100 sequences of 2000 data values.
• 8 real models selected.
• Different lengths of ‘look ahead’ period looked at:
kLimit values = 0 (i.e. no ‘look ahead’ period), 5, 10, 25.
• drequired value kept constant at 5%.
TESTING METHODOLOGY
![Page 18: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/18.jpg)
5 performance measures
1. Coverage of the true mean2. Bias3. Absolute Bias4. Average Nsol value5. Comparison of 4. with Theoretical Nsol
value
• For real models: ‘true’ mean and st.dev values - estimated from whole sets of output data (3000 to 11000 data points).
Microsoft Office Excel Worksheet
![Page 19: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/19.jpg)
Results
• Nsol values for individual algorithm runs are very variable.
• Average Nsol values for 100 runs per model close to the theoretical values of Nsol.
• Normality assumption appears robust.
• Using a ‘look ahead’ period improves performance of the algorithm.
![Page 20: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/20.jpg)
Mean bias significantly different to zero
Failed in coverage of true mean
Mean est. Nsol significantly different to theoretical Nsol (>3)
No ‘look-ahead’ period
Proportion of Artificial models
4/24 2/24 9/18
Proportion of Real models
1/8 1/8 3/5
kLimit = 5 Proportion of Artificial models
1/24 0 1/18
Proportion of Real models
0 0 0
![Page 21: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/21.jpg)
% decrease in absolute mean bias
kLimit = 0 tokLimit = 5
kLimit = 5 tokLimit = 10
kLimit = 10 tokLimit = 25
ArtificialModels
8.76% 0.07% 0.26%
RealModels
10.45% 0.14% 0.33%
Impact of different look ahead periods on performance of algorithm
![Page 22: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/22.jpg)
Number of times the Nsol value changes (out of 100 runs of the algorithm per model) because of the lengthening of the ‘look ahead’ period.
Model ID
kLimit = 0 to kLimit = 5
kLimit = 5 tokLimit = 10
kLimit = 10 to kLimit = 25
R1 0 0 0
R3 2 0 0
R5 24 0 1
R8 24 4 1
A5 30 1 3
A6 26 6 3
A15 1 0 0
A17 22 0 1
A21 25 2 1
A24 37 0 0
![Page 23: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/23.jpg)
Model ID kLimit Nsol Theoretical Nsol (approx)
Mean estimate significantly different to the true mean?
A22 0 4 64 Yes
5 54 No
A9 0 4 112 Yes
5 120 No
A24 0 3 755 Yes
5 718 No
A21 0 8 37 Yes
5 38 No
R7 0 3 10 Yes
5 8 No
R4 0 3 6 Yes
5 7 No
R8 0 3 45 Yes
5 46 No
Eg.s of changes in Nsol & improvement in estimate of true mean
![Page 24: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/24.jpg)
Model ID
kLimit Nsol Theoretical Nsol (approx)
Mean estimate significantly different to the true mean?
A9 0 4 112 Yes
5 120 No
A24 0 3 755 Yes
5 718 No
R7 0 3 10 Yes
5 8 No
R4 0 3 6 Yes
5 7 No
R8 0 3 45 Yes
5 46 No
Examples of changes in Nsol & improvement in estimate of true mean
![Page 25: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/25.jpg)
DISCUSSION
• kLimit default value set to 5.
• Initial number of replications set to 3.
• Multiple response variables - Algorithm run with each response - use maximum estimated value for Nsol.
• Different scenarios - advisable to repeat algorithm every few scenarios to check that precision has not degraded significantly.
• Inclusion into simulation package: Full explanations of algorithm and results.
![Page 26: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/26.jpg)
SUMMARY
• Selection and automation of Confidence Interval (with Specified Precision) Method for estimating the number of replications to be run in a simulation.
• Algorithm created with ‘look ahead’ period -efficient and performs well on wide selection of artificial and real model output.
• ‘Black box’ - fully automated and does not require user intervention.
![Page 27: The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d5c5503460f94a3b2ef/html5/thumbnails/27.jpg)
ACKNOWLEDGMENTSThis work is part of the Automating Simulation Output
Analysis (AutoSimOA) project (http://www.wbs.ac.uk/go/autosimoa) that is funded by the UK Engineering and Physical Sciences Research Council
(EP/D033640/1). The work is being carried out in collaboration with SIMUL8 Corporation, who are also
providing sponsorship for the project.
Katy Hoad, Stewart Robinson, Ruth DaviesWarwick Business School
OR49 Sept 07