Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48,...
-
date post
21-Dec-2015 -
Category
Documents
-
view
221 -
download
0
Transcript of Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48,...
![Page 1: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/1.jpg)
Automating the Analysis of Simulation Output Data
Stewart Robinson, Katy Hoad, Ruth Davies
OR48, September 2006
![Page 2: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/2.jpg)
Outline
The problem
A prototype automated output Analyser
Findings from prototype Analyser
The AutoSimOA Project
Current work - Collecting and characterising real and artificial models
![Page 3: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/3.jpg)
The Problem
Prevalence of simulation software: ‘easy-to-develop’ models and use by non-experts.
Simulation software generally have very limited facilities for directing/advising on simulation experiments.
Main exception is directing scenario selection through ‘optimisers’.
With a lack of the necessary skills and support, it is highly likely that simulation users are using their models poorly.
![Page 4: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/4.jpg)
The Problem
Despite continued theoretical developments in simulation output analysis, little is being put into practical use.
There are 3 factors that seem to inhibit the adoption of output analysis methods:
• Limited testing of methods• Requirement for detailed statistical knowledge• Methods generally not implemented in simulation
software (AutoMod/AutoStat is an exception)
A solution would be to provide an automated output ‘Analyser’.
![Page 5: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/5.jpg)
A Prototype Analyser
Simulationmodel
Warm-upanalysis
Run-lengthanalysis
Replicationsanalysis
Use replicationsor long-run?
Recommendationpossible?
Recommend-ation
Output data
Analyser
Obt
ain
mor
e ou
tput
dat
a
Masters Project (3 students).
The Analyser looked at:
• Warm-up
• Run-length
• Number of replications
Scenario analysis could be added.
![Page 6: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/6.jpg)
A Prototype Analyser
A prototype Analyser has been developed in Microsoft Excel.
At present it links to the SIMUL8 software, but it could be used with any software that can be controlled from Excel VBA.
![Page 7: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/7.jpg)
Illustration: Warm-up
Load Analyser into Excel.
Enter name of SIMUL8 model.
Specify initial number of replications and run-length to use.
![Page 8: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/8.jpg)
Welch's Method: Plot of Moving Average (Window = 12 )
0.00
20.00
40.00
60.00
80.00
100.00
120.00
1 84 167 250 333 416 499 582 665 748 831 914
Observation
Mo
vin
g a
vera
ge
Illustration: Warm-up
![Page 9: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/9.jpg)
Illustration: Replications
![Page 10: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/10.jpg)
Findings from Prototype Analyser
It is possible to link an Automated Analyser in Excel to a simulation software tool.
This was just a proof of concept.
Key issues to address:• More thorough testing of output analysis methods for their
accuracy and their generality.• Adaptation of methods to sequential procedures and to
minimise the need for user intervention.
![Page 11: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/11.jpg)
A 3 year, EPSRC funded project (GR EP/D033640/1) in collaboration with SIMUL8 Corporation.
The AutoSimOA Project
Objectives
• To determine the most appropriate methods for automating simulation output analysis
• To determine the effectiveness of the analysis methods• To revise the methods where necessary in order to
improve their effectiveness and capacity for automation• To propose a procedure for automated output analysis of
warm-up, replications and run-length
Only looking at analysis of a single scenario
![Page 12: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/12.jpg)
The AutoSimOA Project
CURRENT WORK:
1. Literature review of warm-up, replications and run-length methods.
2. Development of artificial data sets (Auto-Regressive; Moving average; M/M/n/p Queues…)
3. Collection of ‘real’ simulation models.
![Page 13: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/13.jpg)
Use models / data sets:
Provide a representative and sufficient set of models / data output for use in discrete event simulation research.
Use models / data sets to test the chosen simulation output analysis methods in the AutoSimOA Project..
![Page 14: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/14.jpg)
Categorising Output Data Sets by Shape & Characteristics
Group A
…Group NGroup B
Auto Correlation Spread round mean
NormalityTrend
Cycling/Seasonality
Terminating
Non-terminating
Steady state
In/out of control
Transient
![Page 15: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/15.jpg)
Model characteristics
Deterministic or random
Significant pre-determined model changes (by time)
Dynamic internal changes i.e. ‘feed-back’
Empty-to-empty pattern
Initial transient (warm-up)
Out of control trend ρ≥1
Cycle
Auto-correlation
Statistical distribution
Output data characteristics
![Page 16: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/16.jpg)
ARTIFICIAL MODELS
Create simple models where theoretical value of some attribute is known.
E.g. M/M/1: mean waiting time.
Create simple models where value of some attribute is estimated but model characteristics can be controlled.
E.g. Single item inventory management system: Number-in-stock.
Construct output, which closely resembles real model output, with known value for some specific attribute.
E.g. AR(1) with Normal errors
Create different output types Transient
Steady state
Steady state cycle
Trend + Initial transient (warm-up)
![Page 17: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/17.jpg)
Example artificial models:
1. Auto-Regressive (2) series
BiasFnXXX ttt 21 5.025.0
1005.010 te
220010 1005.0 t
Sine t
Exponential
Under Damped oscillations
Mean shift 2
Initial Bias Functions:
![Page 18: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/18.jpg)
Run1 ~ AR(2) + "underdamped oscillations" initial bias
-30
-20
-10
0
10
20
30
40
0 100 200 300 400 500
t
Run1 ~ AR(2) + "mean shift" initial bias
-6
-4
-2
0
2
4
6
8
10
12
14
0 100 200 300 400 500t
Run1 ~ AR(2) + "exponential" initial bias
-10
-5
0
5
10
15
20
25
30
35
40
0 100 200 300 400 500t
Run1 ~ AR(2) with no initial bias
-4
-3
-2
-1
0
1
2
3
4
5
0 100 200 300 400 500
t
![Page 19: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/19.jpg)
mean 1.8
Example artificial models:
2. E4 ~ Erlang(4) / M / 1 Queue
Traffic Intensity = 0.8
Queuing time for each customer in a E4/M/1Queuing System
0
2
4
6
8
10
12
14
16
1 432 863 1294 1725 2156 2587 3018 3449 3880 4311 4742
index
![Page 20: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/20.jpg)
REAL MODELS
Models created in “real circumstances” that cover each
general type of model and output encountered in real life modeling.
e.g. Call centre: percentage of calls answered within 30
secs
e.g. Production Line Manufacturing Plant:
through-put / hour
e.g. Fast Food Store:
average queuing timee.g. Swimming
Pool complex: average
number in system
TransientSteady State
Cycle
Steady State
With or without warm-up
Trend
![Page 21: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/21.jpg)
0
2
4
6
8
10
12
14
16
18
0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 320 340 360 380 400 420 440 460 480
time (mins)
num
ber
of custo
mers
0
5
10
15
20
25
30
35
40
45
0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 320 340 360 380 400 420 440 460 480
time (mins)
num
ber
of custo
mers
Example ‘ real ’ models:1. Argos – Number of customers in queue to pay
Stochastic model with changing arrival rates.
Empty to empty; transient; autocorrelated; non-normal output.
![Page 22: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/22.jpg)
0
50
100
150
200
Time (hours)
Num
ber
of ite
ms
Example ‘ real ’ models:2. Leggings Manufacturing Plant – Through-put / hour
Stochastic model.
Steady state with warm-up; not autocorrelated; normal output.
0
50
100
150
200
Time (hours)
Num
ber
of ite
ms
![Page 23: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/23.jpg)
0
10
20
30
40
50
60
1 5 9 13 17 21 25 29 33 37 41
Time (hours)
Num
ber
of
com
ple
ted ite
ms
Example ‘ real ’ models:3. Sanitory Towel Packing Plant – Through-put / hour
Stochastic model with changing productivity in work stations.
Steady state daily cycle.
40
41
42
43
44
45
1 10 19 28 37 46 55 64 73 82 91 100 109 118
Time (days)
Mea
n th
roug
h-pu
t for
the
day
Series of means of each cycle:
autocorrelated; non-normal output.
![Page 24: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/24.jpg)
Use this representative and sufficient set of models/outputwhen
The AutoSimOA Project
• determining the most appropriate methods for automating simulation output analysis
• determining the effectiveness of the analysis methods
• revising the methods where necessary in order to improve their effectiveness and capacity for automation
In order to propose a procedure for automated output analysis of warm-up, replications and run-length.
![Page 25: Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006.](https://reader034.fdocuments.net/reader034/viewer/2022051415/56649d5c5503460f94a3b3c1/html5/thumbnails/25.jpg)
Automating the Analysis of Simulation Output Data
Stewart Robinson, Katy Hoad, Ruth Davies
OR48, September 2006