Statistical Tests for Multiple Forecast...
Transcript of Statistical Tests for Multiple Forecast...
![Page 1: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/1.jpg)
Statistical Tests for Multiple Forecast Comparison
Roberto S. Mariano(Singapore Management University & University of Pennsylvania)
Daniel Preve(Uppsala University)
June 6-7, 2008T.W. Anderson Conference, Stanford University
![Page 2: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/2.jpg)
Why Test for Predictive Ability?
![Page 3: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/3.jpg)
Tests for Equal Predictive Accuracy
![Page 4: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/4.jpg)
The Diebold-Mariano (DM) Test
![Page 5: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/5.jpg)
Introductory Remarks
• Obvious desirability of formal testing procedures
• But earlier efforts at assessing forecast accuracy revolved around calculation of summary error statistics-mainly due to complexities in dealing with sampling uncertainties and correlations present in forecast errors
![Page 6: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/6.jpg)
Introductory Remarks ... continue
• Formal testing approaches started with loss functions that are quadratic in forecast errors; & forecast errors are assumed to be Gaussian and serially uncorrelated
• More recent efforts – much more relaxed conditions– Loss functions may be nonquadratic and asymmetric– Forecast errors need not be Gaussian– Generally based on large-sample asymtotic analysis– With limited experimental studies on small-sample
properties
![Page 7: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/7.jpg)
Significance Tests of Forecast Accuracy
• Model-based tests– Assumes an econometric model, typically
parametric– Model is estimated from a given data sample– Data and model are both available for testing
forecast accuracy– Applied in large macroeconometric models,
using deterministic and stochastic simulations of the estimated model
![Page 8: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/8.jpg)
Significance Tests of Forecast Accuracy ... continue
• Model-free tests– Limited information set: set of forecasts and
actual values of the predictand
![Page 9: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/9.jpg)
Preliminaries (1)
• Available information: t=1,2,3, … T– Actual values yt
– Forecast i: ŷit, i=1,2• Forecast errors: eit = ŷit - yt
• Loss depends on forecast and actual values only through the forecast error:
g(yt, ŷit) = g(ŷit - yt) = g(eit) • Loss differential between the two forecasts
d(t) = g(e1t) – g(e2t)
![Page 10: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/10.jpg)
Preliminaries (2)
• Two forecasts have equal accuracy if and only if the loss differential has zero expectation for all t
• Hence, test H0: E(dt) = 0 for all t
versus the alternative hypothesis H1: E(dt) = , different from zeroμ
![Page 11: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/11.jpg)
Morgan-Granger-Newbold (MGN) Test (1977)
• AssumeA(1) Loss is quadraticA(2) Forecast errors are (a) zero mean, (b)
Gaussian, ( c ) serially uncorrelated• Let
xt = e1t + e2tzt = e1t – e2t
• Here, H0 is equivalent to equality of the two forecast error variances, or, equivalently, zero correlation between xt and zt
![Page 12: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/12.jpg)
Variations of MGN Test
• Harvey, Leybourne and Newbold (1997) regression set up
xt = β zt + εt
• The MGN test statistic is exactly the same as that for testing the null hypothesis that β = 0 in this regression.
![Page 13: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/13.jpg)
Variations of MGN Test ... continue
• When the forecast errors come from a heavy-tailed distribution , HLN argue that the estimate of the variance of b is biased and suggest utilizing a White-correction for heteroskedasticityto estimate the variance of b.
• Another HLN variation: Spearman’s rank test for zero correlation between x and z
![Page 14: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/14.jpg)
Variations of MGN Test ... continue
• Real drawback of all these tests: limitation of applicability to one-step predictions and to squared error loss
![Page 15: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/15.jpg)
Meese-Rogoff (MR) Test (1988)
• Now, forecast errors can be serially and contemporaneously correlated
• Still maintain assumptions A1, A2a, and A2b and assume squared error loss
• The MR test is based on the sample covariance between xt and zt
![Page 16: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/16.jpg)
Diebold-Mariano (DM) Test (1995)
• Applicable to nonquadratic loss functions, multi-period forecasts, and forecast errors that are non-Gaussian, nonzero-mean, serially correlated, and contemporaneously correlated.
• Basis of the test: sample mean of the observed loss differential series– {dt : t=1, 2, …}
![Page 17: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/17.jpg)
DM Test (2)• Assuming covariance stationarity and other
regularity conditions on the process {dt}, thenconverges in distribution to
N(0, 2 π fd(0)),
• fd(.) is the spectral density of {dt}
• is the sample mean loss differential
1/2 ( )T d μ−
d
![Page 18: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/18.jpg)
DM Test Statistic
where is a consistent estimate
of fd(0).
1/2ˆ/ [2 (0) / ]dDM d f Tπ=
ˆ (0)df
![Page 19: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/19.jpg)
Small-Sample Modification of DM Test
• HLN (1997) :use an approximately unbiased estimate of the variance of the mean loss differential
• Forecast accuracy is measured in terms of mean squared prediction error
![Page 20: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/20.jpg)
Small-Sample Modification of DM Test ... continue
• H-step ahead forecast errors are assumed to have zero autocorrelations at order h and beyond
• Small-sample modification
DM* = DM/{[T+1-2h+h(h-1)/T]/T}1/2
t-distribution with T-1 . .d f
![Page 21: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/21.jpg)
Applications
• Predictability of nominal exchange rates (Mark 1995)
• Comparing predictive ability of flexible-specification, fixed-specification, linear and nonlinear econometric models of macroeconomic variables (Swanson & White 1997)
![Page 22: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/22.jpg)
Applications
• Predictive ability with cointegrated variables (Corradi, Swanson & Olivetti 2001)
• Predictive ability in the presence of structural breaks (Clark & McCracken 2003)
• Forecast comparison of volatility models versus GARCH (1,1) – (Hansen & Lunde 2005)
![Page 23: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/23.jpg)
Applications
• Forecast comparison of volatility models versus GARCH (1,1) – (Hansen & Lunde2005)
![Page 24: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/24.jpg)
A Multivariate Test
![Page 25: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/25.jpg)
Invariance and Bias
![Page 26: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/26.jpg)
Two Modified Tests
![Page 27: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/27.jpg)
Monte Carlo Setup
![Page 28: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/28.jpg)
![Page 29: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/29.jpg)
Multivariate CaseMonte Carlo Results
• The proposed test can be oversized in moderate samples
• The test benefits noticeably from the finite-sample correction, even in moderately large samples
• However, the finite-sample correction provides only a partial adjustment
![Page 30: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/30.jpg)
Multivariate CaseFollow-up Work
• Consider alternative types of weak stationarity• Extensions to
– Panel data (Pesaran)– High frequency data– Qualitative and limited dependent variable– Semiparametric approaches
• Compare with White / Hansen's data snooping reality test
• Relation to Ken West’s test for predictive ability• Semiparametric approaches to multivariate tests of
forecasting performance• Power considerations
![Page 31: Statistical Tests for Multiple Forecast Comparisonstatweb.stanford.edu/~ckirby/ted/conference/Roberto Mariano.pdf · Statistical Tests for Multiple Forecast Comparison Roberto S.](https://reader033.fdocuments.net/reader033/viewer/2022051600/5a9ef80b7f8b9a6c178c1e60/html5/thumbnails/31.jpg)
The End