An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting...
Transcript of An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting...
![Page 1: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/1.jpg)
An Introduction to the PRObabilityFORecasting (PROFOR) Toolbox for
MATLAB
Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)and Shaun Vahey (Warwick)
Narodowy Bank Polski
November 2014
Presentation by Shaun Vahey (Warwick) PROFOR 1/22
![Page 2: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/2.jpg)
What is PROFOR?
I PROFOR provides macro models, estimation algorithms,techniques for forecast combination, forecast evaluationtools and real-time data suitable for short-term quarterly(or monthly) predictions
I Trial version (TBR via WBS website, Dec 2014) packagestogether many best practice forecasting methods
I A network of practitioners and academics supports projectand trial version of PROFOR
Presentation by Shaun Vahey (Warwick) PROFOR 2/1
![Page 3: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/3.jpg)
Summary of application
I UK macroeconomic forecasting application based on the(potentially) asymmetric predictive densities published bythe Bank of England for inflation
I Forecast evaluation methods include examining calibration,predictive scores, and decision-theoretic forecast evaluation
I How much better is the BOE approach (one step ahead)than, say, an AR(1) benchmark?
I Tomorrow’s session replicates analysis in live PROFORsession and provides further examples (models estimated,forecasts combined, then evaluated)
Presentation by Shaun Vahey (Warwick) PROFOR 3/1
![Page 4: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/4.jpg)
What does PROFOR aim to do?
I Platform to produce real-time macro density forecasts
I Easy to modify/scalable, object oriented, well documented,examples, manual etc.
I No commercial value, released as freeware via WBSwebsite; released under the GNU v 3 license
Presentation by Shaun Vahey (Warwick) PROFOR 4/1
![Page 5: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/5.jpg)
What’s in PROFOR?
I Models: ARs, VARs, IMA, TVP and regime switchingVARs (plus stochastic volatility variants, Bayesiansimulation options)
I Forecasts: iterative, quarterly and monthly
I Combinations: equal, log score, and CRPS weights forLinear Opinion Pools
I Evaluations: log scores, Brier scores and decomposition,CRPS, PITS (with one-shot tests), simple loss functions
I Data: real-time data (from FRED and/or xls), BOE andNB forecast data, external (e.g. user’s) forecast densities
Presentation by Shaun Vahey (Warwick) PROFOR 5/1
![Page 6: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/6.jpg)
Who’s in the PROFOR team?
I Researchers: Craig and Leif Anders; Shaun Vahey, LizWakerly and Anthony Garratt
I Partner investigators: Michael Smith (Melbourne), Simonvan Norden (HEC Montreal and CIRANO), RodneyStrachan (Queensland), Domenico Giannone (ULB)
I Advisory board: Francesco Ravazzolo (NB) and SimonPrice (BOE)
I Partner research centres: CAMP, KOF, CIRANO, CAMA,Veissmann Research Centre, RPF at GWU
I Partner orgs: BOE, NB and WBS
Presentation by Shaun Vahey (Warwick) PROFOR 6/1
![Page 7: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/7.jpg)
PROFOR timeline
I Sep-Nov 2014 trial version of toolbox preview to NB andBOE (feedback’d help!)
I Dec 2014 trial toolbox release on WBS website
I Mar 2015 Documentation for trial version complete (2papers, plus user manual)
I Apr 2015 Next phase begins (non-Gaussian modelling,non-linear methods, mixed frequency)
Presentation by Shaun Vahey (Warwick) PROFOR 7/1
![Page 8: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/8.jpg)
An Example: BOE fan charts
I Two “experts”, BOE and AR(1); see Gneiting and Ranjan(2011, JBES)
I Look at a bake-off between the two experts 2004Q1through 2013Q4 consider calibration (via PITS), log scoresand cost-loss ratio
I In this example, the policymaker would prefer BOE fancharts to call inflation events one step ahead? How muchbetter is it though?
Presentation by Shaun Vahey (Warwick) PROFOR 8/1
![Page 9: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/9.jpg)
Central banks and probabilistic forecasts
I “Given . . . (the) asymmetric costs or benefits of particularoutcomes, a central bank needs to consider not only themost likely future path for the economy, but also thedistribution of possible outcomes about that path. Thedecision-makers then need to reach a judgment about theprobabilities, costs, and benefits of the various possibleoutcomes . . .”
Greenspan (2004, p 37).
Presentation by Shaun Vahey (Warwick) PROFOR 9/1
![Page 10: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/10.jpg)
A BOE perspective
I To the Governor, exceeding the inflation target (even in theshort term) results in a substantial economic cost
I If inflation deviates from central target (2 percent) by morethan 1 percentage point, the Governor must to send anopen letter to the Chancellor
I The Governor sent letters to Chancellor between April2007 and February 2012, all gave reasons for high inflationin short term; preceding Inflation Report forewarned of thetarget breach in each case
Presentation by Shaun Vahey (Warwick) PROFOR 10/1
![Page 11: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/11.jpg)
An inflation event warning
I For example, the Governor’s “Opening Remarks” to theBank of England’s Inflation Report Press Conference inFebruary 2010 stressed
“The January figure for CPI inflation is likely to haveexceeded 3% . . . This would be the third episode wheninflation has temporarily moved above the target . . .requiring me to write an open letter to the Chancellor.”
Presentation by Shaun Vahey (Warwick) PROFOR 11/1
![Page 12: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/12.jpg)
Cost-loss approach with high inflation events, πt > π̄
I Following eg Granger and Pesaran (2000), Berrocal et al(2010), relative cost of unanticipated inflation R = C/L,0 < R < 1, unknown
I Issue (1-step ahead) inflation event warning iffPr(πt > π̄) > R
I Define TEL = n10L + (n01 + n00)CEvent Observed
Event Forecast Yes NoYes n00 n01
No n10 n11
Presentation by Shaun Vahey (Warwick) PROFOR 12/1
![Page 13: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/13.jpg)
![Page 14: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/14.jpg)
![Page 15: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/15.jpg)
![Page 16: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/16.jpg)
![Page 17: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/17.jpg)
![Page 18: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/18.jpg)
![Page 19: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/19.jpg)
![Page 20: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/20.jpg)
![Page 21: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/21.jpg)
![Page 22: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/22.jpg)
![Page 23: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/23.jpg)
![Page 24: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/24.jpg)
Some questions of interest
III Can log score based evaluations of forecast densities maskpredictive content? Yep: log score differentials aren’tsufficient to indicate that policymaker can use expert inreal time to give an early warning indicator
I What additional steps—beyond log scores—might be usefulto analyse forecast performance? Some tricks fromPROFOR toolbox ([email protected])
Presentation by Shaun Vahey (Warwick) PROFOR 19/1
![Page 25: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/25.jpg)
Toolbox tricks
I CRPS Hersbach (2000), Ravazzolo-Vahey (2009, 2013)maximising sharpness conditional on calibration; plusthreshold scoring rules, Gneiting-Ranjan (2011),Garratt-Mitchell-Vahey (2013)
I Analyse PITS to check calibration (reliability) egDiebold-Gunther-Tay (1998), Jore-Mitchell-Vahey (2010);plus resolution vNorden-Galbraith (2008)
I Utilise loss function (if you know it!) Granger-Pesaran(2000a, 2000b)
Presentation by Shaun Vahey (Warwick) PROFOR 20/1
![Page 26: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/26.jpg)
In summary
I Example shows that in terms of one step ahead forecasts,the Bank’s forecasts are substantially better for earlywarning signals of inflationary pressures. We can quantifythe magnitude using a loss function.
I But, there are issues. PITS reveal forecast densities are toodiffuse.
I AR(1) benchmark too narrow (usually, and lagging),implying gains from combination
Presentation by Shaun Vahey (Warwick) PROFOR 21/1
![Page 27: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/27.jpg)
−2 −1 0 1 2 30
0.5
1
1.5
2
2.5
3
3.5(a) h=1, Outturn=−0.0045596
−2 −1 0 1 2 30
0.5
1
1.5
2
2.5
3
3.5(b) h=2, Outturn=0.09753
−2 −1 0 1 2 30
0.5
1
1.5
2
2.5
3(c) h=4, Outturn=0.2422
−2 −1 0 1 2 30
0.5
1
1.5
2
2.5(d) h=8, Outturn=0.61672
BVAR−SV
BVAR
C−Skt
C−EDF
Figure 5: Forecast densities of inflation from using vintage 2009:Q2 data with forecast origin2009:Q1. Panel (a) is for h = 1 quarter ahead (ie. the nowcast), panel (b) for h = 2quarters ahead (2009:Q3), panel (c) if for h = 4 quarters ahead (2010:Q1), and panel (d)is for h = 8 quarters ahead (2011:Q1). Forecast densities are from the BVAR (blue line),BVAR-SV (black dashed line), Coupla & EDF margins (blue dashed line) and Copula &Skew t margins (black line). The outturn values for each quarter are also reported.
35
![Page 28: An Introduction to the PRObability FORecasting …...An Introduction to the PRObability FORecasting (PROFOR) Toolbox for MATLAB Craig Thamotheram (Warwick), Leif Anders Thorsrud (BI)](https://reader033.fdocuments.net/reader033/viewer/2022052423/5f08cfcf7e708231d423d676/html5/thumbnails/28.jpg)
PROFOR: slick, but not for quacks . . .?
I John Kay (FT, September 21 2010):
“There will always be a demand for forecasts,so there will always be a supply. But thereputation of economic forecasters, like otherquacks and charlatans, depends more on theslickness of their presentations than the valueof their work”
Presentation by Shaun Vahey (Warwick) PROFOR 22/1