Confidence Interval Estimation in System Dynamics Models
Gokhan Dogan*
MIT Sloan School of Management
System Dynamics Group
*Special thanks to John Sterman for his support
Motivation
Calibration
Manual Calibration Automated Calibration (e.g. Vensim, Powersim)
Automated Calibration
02468
10121416
Time
Actual Data
Model Output
Motivation
• Once model parameters are estimated with automated calibration,
next step: Estimate confidence intervals!
• Questions:
-Are there available tools at software packages?
-Do these methods have any limitations?
-Are there alternative methods?
Why are confidence intervals important
Parameter Estimate
θ
95% Confidence
Interval
0 Parameter Estimate
θ
95% Confidence
Interval
0
We reject the claim that the parameter value is equal to 0
(with 95% probability)
We can’t reject the claim that the parameter value is equal
to 0 (with 95% probability)
How can we estimate confidence intervals?
Likelihood Ratio Method
Bootstrapping
Used in the System Dynamics Software (Vensim) /Literature
The method we suggest for System Dynamics models
Both methods yield approximate confidence intervals!
Likelihood Ratio Method
• The likelihood ratio method is used in system dynamics software packages (Vensim) and literature (Oliva and Sterman, 2001).
• It relies on asymptotic theory (large sample assumption).
However
Likelihood Ratio Method (as it is used at software packages) assumes:
At system dynamics models:
-Large Sample -It is not always possible to have large sample
-No feedback (autocorrelation)
-There are many feedback loops
-Normally distributed error terms
-Error terms are not always normally distributed
Bootstrapping
• Introduced by Efron (1979) and based on resampling. Extensive survey in Li and Maddala (1996).
• It seems more appropriate for system dynamics models because- It doesn’t require large sample- It is applicable when there is feedback (autocorrelation) - It doesn’t assume normally distributed error terms
Drawbacks of bootstrapping
• The software packages do not implement it.
• It is time consuming.
Bootstrapping
ERROR TERMS
8
4
0
-4
-8
1 13 25 36 48Time (Week)
Error Terms : 15289pres
MODEL_vs_HISTORICAL_DATA
20
15
10
5
0
1 13 25 36 48Time (Week)
Historical Data : 15289pres casesModel Output : 15289pres cases
Compute the Error Terms
Fit the model and estimate parameters
Bootstrapping uses resampling
ERROR TERMS
8
4
0
-4
-8
1 13 25 36 48Time (Week)
Error Terms : 15289pres
Nonparametric: Reshuffle Them and
Generate many many new error term sets
using the autocorrelation information
Parametric: Fit a distribution and
Generate many many new error term sets
using the autocorrelation and distribution information
Resampling the Error Terms
• If we know that:- The error terms are autocorrelated- Their variance is not constant (heteroskedasticity)- They are not normally distributed=> We can use this information while resampling the error terms
• Flexibility of bootstrapping stems from this stage
ERROR TERMS
8
4
0
-4
-8
1 13 25 36 48Time (Week)
Error Terms : 15289pres
ERROR TERMS
20
10
0
-10
-20
1 13 25 36 48Time (Week)
Residuals : subject1_ar1_inf_btstrp5
ERROR TERMS
6
3
0
-3
-6
1 13 25 36 48Time (Week)
Residuals : subject1_ar1_inf_btstrp4
ERROR TERMS
8
4
0
-4
-8
1 13 25 36 48Time (Week)
Residuals : subject1_ar1_inf_btstrp2
. . .
MODEL OUTPUT
20
15
10
5
0
1 13 25 36 48Time (Week)
Model Output : 15289pres cases
HISTORICAL DATA
20
15
10
5
0
1 13 25 36 48Time (Week)
Actual Orders : subject1_ar1_inf_btstrp2
HISTORICAL DATA
20
15
10
5
0
1 13 25 36 48Time (Week)
Actual Orders : subject1_ar1_inf_btstrp4
HISTORICAL DATA
20
15
10
5
0
1 13 25 36 48Time (Week)
Actual Orders : subject1_ar1_inf_btstrp5
. . .
+ =
FABRICATED ERROR TERMS
FABRICATED “HISTORICAL” DATA
HISTORICAL DATA
20
15
10
5
0
1 13 25 36 48Time (Week)
Actual Orders : subject1_ar1_inf_btstrp2
HISTORICAL DATA
20
15
10
5
0
1 13 25 36 48Time (Week)
Actual Orders : subject1_ar1_inf_btstrp4
HISTORICAL DATA
20
15
10
5
0
1 13 25 36 48Time (Week)
Actual Orders : subject1_ar1_inf_btstrp5
. . .
FABRICATED “HISTORICAL” DATA
Fit the model and estimate parameters
Fit the model and estimate parameters
Fit the model and estimate parameters
Parameter Estimate
Parameter Estimate
Parameter Estimate
500
Parameter Estimates
Distribution of a model parameter
Experiments
• We had experimental time series data from 240 subjects.
• Subjects were beer game players.
• For each subject we had 48 data points, so we estimated parameters and confidence intervals using 48 data points.
Model (Same as Sterman 1989)
• Ot = Max[0, θLRt + (1–θ)ELt + α(S' – St –βSLt) + error termt]
Parameters to be estimated are θ, α, β, S‘
Individual Results
θ=0.95
10
95% Confidence Intervals for θ
10
θ=0.95
0.77
0.01
95% CI
95% CI
Likelihood Ratio Method
Bootstrapping
Individual Results
β =0.01
0.20
95% Confidence Intervals for β
95% CI
Likelihood Ratio Method
Bootstrapping
β =0.01
0.20 95% CI
Significantly Different From 0!!!
Overall Results
Theta Alpha Beta S-Prime
Likelihood Ratio Method 0.19 0.11 0.11 13.20
Bootstrapping 0.67 0.30 0.52 973.59
Average 95% Confidence Interval Length
Theta Alpha Beta S-Prime
Likelihood Ratio Method 0.10 0.08 0.06 2.32
Bootstrapping 0.84 0.24 0.48 10.10
Median of 95% Confidence Interval Length
Overall Results
Theta Alpha Beta S-Prime
Bootstrapping CI wider than Likelihood Ratio Method CI
97.76% 98.81% 100% 98.56%
Percentage of Subjects for whom the bootstrapping confidence interval is wider than the likelihood ratio method confidence interval
Likelihood Ratio Method vs Bootstrapping
• Likelihood Ratio Method: Is easy to compute Very fast BUT depends on
assumptions that are usually violated by system dynamics models
Yields very tight confidence intervals
• Bootstrapping: Is NOT easy to compute Takes longer time DOES NOT depend on
assumptions that are usually violated by system dynamics models
Yields larger confidence intervals. Usually more conservative.
Top Related