Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of...
-
date post
20-Dec-2015 -
Category
Documents
-
view
226 -
download
2
Transcript of Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of...
![Page 1: Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d4e5503460f94a2dd08/html5/thumbnails/1.jpg)
Confidence Interval Estimation in System Dynamics Models
Gokhan Dogan*
MIT Sloan School of Management
System Dynamics Group
*Special thanks to John Sterman for his support
![Page 2: Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d4e5503460f94a2dd08/html5/thumbnails/2.jpg)
Motivation
Calibration
Manual Calibration Automated Calibration (e.g. Vensim, Powersim)
Automated Calibration
02468
10121416
Time
Actual Data
Model Output
![Page 3: Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d4e5503460f94a2dd08/html5/thumbnails/3.jpg)
Motivation
• Once model parameters are estimated with automated calibration,
next step: Estimate confidence intervals!
• Questions:
-Are there available tools at software packages?
-Do these methods have any limitations?
-Are there alternative methods?
![Page 4: Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d4e5503460f94a2dd08/html5/thumbnails/4.jpg)
Why are confidence intervals important
Parameter Estimate
θ
95% Confidence
Interval
0 Parameter Estimate
θ
95% Confidence
Interval
0
We reject the claim that the parameter value is equal to 0
(with 95% probability)
We can’t reject the claim that the parameter value is equal
to 0 (with 95% probability)
![Page 5: Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d4e5503460f94a2dd08/html5/thumbnails/5.jpg)
How can we estimate confidence intervals?
Likelihood Ratio Method
Bootstrapping
Used in the System Dynamics Software (Vensim) /Literature
The method we suggest for System Dynamics models
Both methods yield approximate confidence intervals!
![Page 6: Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d4e5503460f94a2dd08/html5/thumbnails/6.jpg)
Likelihood Ratio Method
• The likelihood ratio method is used in system dynamics software packages (Vensim) and literature (Oliva and Sterman, 2001).
• It relies on asymptotic theory (large sample assumption).
![Page 7: Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d4e5503460f94a2dd08/html5/thumbnails/7.jpg)
However
Likelihood Ratio Method (as it is used at software packages) assumes:
At system dynamics models:
-Large Sample -It is not always possible to have large sample
-No feedback (autocorrelation)
-There are many feedback loops
-Normally distributed error terms
-Error terms are not always normally distributed
![Page 8: Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d4e5503460f94a2dd08/html5/thumbnails/8.jpg)
Bootstrapping
• Introduced by Efron (1979) and based on resampling. Extensive survey in Li and Maddala (1996).
• It seems more appropriate for system dynamics models because- It doesn’t require large sample- It is applicable when there is feedback (autocorrelation) - It doesn’t assume normally distributed error terms
![Page 9: Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d4e5503460f94a2dd08/html5/thumbnails/9.jpg)
Drawbacks of bootstrapping
• The software packages do not implement it.
• It is time consuming.
![Page 10: Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d4e5503460f94a2dd08/html5/thumbnails/10.jpg)
Bootstrapping
ERROR TERMS
8
4
0
-4
-8
1 13 25 36 48Time (Week)
Error Terms : 15289pres
MODEL_vs_HISTORICAL_DATA
20
15
10
5
0
1 13 25 36 48Time (Week)
Historical Data : 15289pres casesModel Output : 15289pres cases
Compute the Error Terms
Fit the model and estimate parameters
![Page 11: Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d4e5503460f94a2dd08/html5/thumbnails/11.jpg)
Bootstrapping uses resampling
ERROR TERMS
8
4
0
-4
-8
1 13 25 36 48Time (Week)
Error Terms : 15289pres
Nonparametric: Reshuffle Them and
Generate many many new error term sets
using the autocorrelation information
Parametric: Fit a distribution and
Generate many many new error term sets
using the autocorrelation and distribution information
![Page 12: Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d4e5503460f94a2dd08/html5/thumbnails/12.jpg)
Resampling the Error Terms
• If we know that:- The error terms are autocorrelated- Their variance is not constant (heteroskedasticity)- They are not normally distributed=> We can use this information while resampling the error terms
• Flexibility of bootstrapping stems from this stage
![Page 13: Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d4e5503460f94a2dd08/html5/thumbnails/13.jpg)
ERROR TERMS
8
4
0
-4
-8
1 13 25 36 48Time (Week)
Error Terms : 15289pres
ERROR TERMS
20
10
0
-10
-20
1 13 25 36 48Time (Week)
Residuals : subject1_ar1_inf_btstrp5
ERROR TERMS
6
3
0
-3
-6
1 13 25 36 48Time (Week)
Residuals : subject1_ar1_inf_btstrp4
ERROR TERMS
8
4
0
-4
-8
1 13 25 36 48Time (Week)
Residuals : subject1_ar1_inf_btstrp2
. . .
MODEL OUTPUT
20
15
10
5
0
1 13 25 36 48Time (Week)
Model Output : 15289pres cases
HISTORICAL DATA
20
15
10
5
0
1 13 25 36 48Time (Week)
Actual Orders : subject1_ar1_inf_btstrp2
HISTORICAL DATA
20
15
10
5
0
1 13 25 36 48Time (Week)
Actual Orders : subject1_ar1_inf_btstrp4
HISTORICAL DATA
20
15
10
5
0
1 13 25 36 48Time (Week)
Actual Orders : subject1_ar1_inf_btstrp5
. . .
+ =
FABRICATED ERROR TERMS
FABRICATED “HISTORICAL” DATA
![Page 14: Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d4e5503460f94a2dd08/html5/thumbnails/14.jpg)
HISTORICAL DATA
20
15
10
5
0
1 13 25 36 48Time (Week)
Actual Orders : subject1_ar1_inf_btstrp2
HISTORICAL DATA
20
15
10
5
0
1 13 25 36 48Time (Week)
Actual Orders : subject1_ar1_inf_btstrp4
HISTORICAL DATA
20
15
10
5
0
1 13 25 36 48Time (Week)
Actual Orders : subject1_ar1_inf_btstrp5
. . .
FABRICATED “HISTORICAL” DATA
Fit the model and estimate parameters
Fit the model and estimate parameters
Fit the model and estimate parameters
Parameter Estimate
Parameter Estimate
Parameter Estimate
500
Parameter Estimates
![Page 15: Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d4e5503460f94a2dd08/html5/thumbnails/15.jpg)
Distribution of a model parameter
![Page 16: Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d4e5503460f94a2dd08/html5/thumbnails/16.jpg)
Experiments
• We had experimental time series data from 240 subjects.
• Subjects were beer game players.
• For each subject we had 48 data points, so we estimated parameters and confidence intervals using 48 data points.
![Page 17: Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d4e5503460f94a2dd08/html5/thumbnails/17.jpg)
Model (Same as Sterman 1989)
• Ot = Max[0, θLRt + (1–θ)ELt + α(S' – St –βSLt) + error termt]
Parameters to be estimated are θ, α, β, S‘
![Page 18: Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d4e5503460f94a2dd08/html5/thumbnails/18.jpg)
Individual Results
θ=0.95
10
95% Confidence Intervals for θ
10
θ=0.95
0.77
0.01
95% CI
95% CI
Likelihood Ratio Method
Bootstrapping
![Page 19: Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d4e5503460f94a2dd08/html5/thumbnails/19.jpg)
Individual Results
β =0.01
0.20
95% Confidence Intervals for β
95% CI
Likelihood Ratio Method
Bootstrapping
β =0.01
0.20 95% CI
Significantly Different From 0!!!
![Page 20: Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d4e5503460f94a2dd08/html5/thumbnails/20.jpg)
Overall Results
Theta Alpha Beta S-Prime
Likelihood Ratio Method 0.19 0.11 0.11 13.20
Bootstrapping 0.67 0.30 0.52 973.59
Average 95% Confidence Interval Length
Theta Alpha Beta S-Prime
Likelihood Ratio Method 0.10 0.08 0.06 2.32
Bootstrapping 0.84 0.24 0.48 10.10
Median of 95% Confidence Interval Length
![Page 21: Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d4e5503460f94a2dd08/html5/thumbnails/21.jpg)
Overall Results
Theta Alpha Beta S-Prime
Bootstrapping CI wider than Likelihood Ratio Method CI
97.76% 98.81% 100% 98.56%
Percentage of Subjects for whom the bootstrapping confidence interval is wider than the likelihood ratio method confidence interval
![Page 22: Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d4e5503460f94a2dd08/html5/thumbnails/22.jpg)
Likelihood Ratio Method vs Bootstrapping
• Likelihood Ratio Method: Is easy to compute Very fast BUT depends on
assumptions that are usually violated by system dynamics models
Yields very tight confidence intervals
• Bootstrapping: Is NOT easy to compute Takes longer time DOES NOT depend on
assumptions that are usually violated by system dynamics models
Yields larger confidence intervals. Usually more conservative.