with Monte-Carlo Dropout in Neural Networks · [7]Juan.M Morales, Conejo, and et.al. Integrating...
1
Deep Probabilistic Forecasting with Monte-Carlo Dropout in Neural Networks Novin Shahroudi Institute of Computer Science, University of Tartu [email protected] — github.com/novinsh/master thesis — @novinsh Forecasting is an act of predicting the future that comes with a de- gree of uncertainty. That is why a probabilistic expression of the future suits its remote and uncertain nature better than other forms. Deep Learning is taking a prevalent role in today’s industries. It is being employed in more mission-critical tasks such as self-driving cars as well as tasks that may not require a safety measure, yet require some measure of reliability such as in energy power fore- casting. Taking uncertainty into account in these models is a key enabler to account for safety and reliability in these models. This poster demonstrates some of the experiments performed for my master thesis to perform probabilistic forecasting using a type of Variational Bayesian Approximation for neural networks. Reliable forecasting models play a crucial role in operations of electricity grid. Wind power is expected to provide at least 50% of the electricity production in Estonia and Sweden in the upcoming decade. Therefore, this work was motivated to be applied on Wind Power Forecasting. Background Time-series Forecasting In the Time series forecasting, a history of data up to time is given and a prediction should be made for the future for a given num- ber of steps. Data points are dependent on each other and may not be identically distributed. Each step in the forecast referred to as a lead time, and the whole forecast steps referred to as horizon. Equa- tion (1) denotes the observed values of a time series from the past to the present and the future. y 0 ,y 1 ,...,y t ,y t+1 ,...,y t+h h ∈ N >0 (1) Additional features could be incorporated in univariate time series forecasting that it is referred to as exogenous variables. Bayesian Deep Learning A Bayesian neural network (BNN) [1] considers a distribution over its parameters as well as its outputs as opposed to a generic Artificial Neural Network (ANN) which may be referred to as Non-bayesian Neural Network as well. Calculating the posterior of a Bayesian model is intractable for neural networks due to humongous param- eter space even with relatively small networks. In [3] authors in- troduce Monte-Carlo Dropout (MCDO) network as an approxima- tion of BNN that gives reasonably good results to approximate a Bayesian Neural Network. Probabilistic Forecasting Probabilistic or Dense Forecast is the most informative form of forecast that estimates a probability distribution rather a point fore- cast for each time step. It is common to demonstrate a probabilistic forecast using fan-charts, each fan representing a quantile of the probability distribution as depicted in Fig. 5. Evaluation metrics CRPS [7] evaluates quality of a probabilistic forecast represented by a predictive cumulative distribution function F t (y ) against the true value or observed probability passed through a heaviside function H (y, ˆ y ) as shown in Eq. (2). An important property of CRPS is that it measures the accuracy, as well as the sharpness of the predictions which means calibratedness of the forecasts is also reflected in the score. When CRPS normalized, it is denoted as NCRPS. NCRPS (y )= 1 n(T) X t∈T Z ∞ -∞ ( F t+h|t (y ) - H(y - y t+h ) ) 2 (2) Mean Squared Error (MSE) has the same measurement unit as of the squared of the series evaluated for. It is also equivalent to the variance of the model output if the model is unbiased and it is tra- ditionally used for point estimate forecasts. MSE (h)= 1 n(T) X t∈T (ˆ y t+h|t - y t+h ) 2 (3) Data For the experiments with Synthetic data, the TimeSynth was used to generate and sample from a sinusoidal signal with frequency. For the experiments with real data, wind power dataset from Global Energy Forecasting Competition 2014 [5] was used. Setup of the data performed according to [4]. Two sets of experiments conducted on this dataset, a univariate forecasting, and univariate forecasting with 4 wind speed features as other predictors. Baseline Models Persistent or naive forecast is a well-known time-series forecasting baseline. It uses the last observation as the forecast. Methods Probabilistic Baseline Naive method extended to provide probabilistic baseline where a given quantile of the training data is being used as the forecast. Models Methods explained in this section were mainly inspired by [6]. A Multi-layer Perceptron (MLP), and a Recurrent Neural Network (RNN) with GRU cells [2] as depicted in Fig. 1 were used. Both models provided two outputs, one estimating the mean and the other the variance of the forecast. 100 units were used with a dropout rate of 0.3 for both models. For the RNN, recurrent dropout was used. The batch size for all experiments set to 32. All models were run for 10 epochs with Adam optimizer and Cyclical Learning Rate Sched- uler [8]. Activation function for the first layer set to linear for the MLP and tanh for the RNN. Activation of the first dense output that estimate the mean set to hard sigmoid, and the second dense output that estimates the logarithm of the variance set to linear. Loss1 + Total loss Loss2 Dense 100 flatten Dense 1 Input Dropout Dense 2 Input Dense 1 GRU 100 dropout Dense 1 + Total loss Loss1 Loss2 Architecture of MLP (left) and RNN (right) Models were trained to forecast one-step ahead and to achieve multi-step ahead forecast roll-forward forecasting was used. Uncertainty in the data, also referred to as Aleatoric uncertainty, is mainly caused by partial observability and stochasticity in the underlying process generating the data, and/or errors in measure- ments. It is estimated by the second output of the models. A new loss as shown in Eq. 4 is then used. In this way, the network is en- couraged not only minimize the mean but also the variance of the predictions. L(θ )= 1 N N X i=1 1 σ ( ~ x i ) 2 || ~ y i - f ( ~x i ) || 2 +logσ ( ~ x i ) 2 (4) Uncertainty in the model also referred to as Epistemic uncertainty. It can be caused by model misspecification (a wrong parametric family), error in the estimation of the model’s parameters when modeling, exposing the model to novel examples and any other sys- tematic error in the model. This can be estimated by Dropout at test time. It produces different possible outputs that can be thought to be produced by different sub-networks which would imply the vari- ance of the model. A neural network can also be thought of as an ensemble method for reducing the variance of prediction by using a number of unbiased models. Scenario Forecasting The roll-forward forecasting produces biased results as a result of the model’s ignorance caused by treating forecast values as new ob- servations. To tackle this issue scenario forecasting was used which also enabled the model to produce different forecast trajectories as another informative form of forecasting. Since the forecasts were probabilistic, one could sample from the forecast distribution and the sampled values could be used instead as the next value in the input sequence forecasting for all possible alternative futures. Demonstration of constructing trajectories with scenario forecasting. On the top, possible samples drew from the forecast distribution (denoted by red) in order to perform scenario forecasting on these samples (denoted by blue). On the bottom, generated trajectories as a result of scenario forecasting demonstrated. Scenarios reveal possible multimodality in the data. Each of these scenarios forecasts per- formed as part of a MCDO simulation and so were averaged over all to obtain the final result as shown Figure 5 Results All results obtained by evaluation on 20 different splits of the val- idation set. Input size for the Synthetic data was 24 and of 4 for the GEFCom’14 with forecast horizon of size 48. Results of 5 variation of each model per each architecture are being reported here. 4 of which correspond to the Bayesian models and to a non- Bayesian model (old). The Bayesian models are denoted as AL for Aleatoric, the EP for Epistemic, the ALEP for Aleatoric+Epistemic, and SFALEP for the Scenario Forecast+ALEP. Finally, the proba- bilistic baseline denoted by QNaive. NCRSP results obtained on the GEFCom’14 Dataset NCRPS Univariate with Exogenous Variables Architectures MLP GRU MLP GRU QNaive 0.211 ± 0.103 0.211 ± 0.103 0.209 ± 0.092 0.209 ± 0.092 AL 0.215 ± 0.157 0.218 ± 0.155 0.167 ± 0.121 0.169 ± 0.150 EP 0.247 ± 0.197 0.263 ± 0.186 0.196 ± 0.154 0.157 ± 0.141 ALEP 0.208 ± 0.147 0.217 ± 0.150 0.166 ± 0.097 0.159 ± 0.137 SFALEP 0.203 ± 0.149 0.183 ± 0.097 0.173 ± 0.104 0.144 ± 0.099 Results obtained on the Synthetic Dataset Metrics CRPS MSE Architectures MLP GRU MLP GRU old N/A N/A 0.033 ± 0.020 0.035 ± 0.021 QNaive 0.163 ± 0.055 0.163 ± 0.055 0.082 ± 0.014 0.082 ± 0.014 AL 0.116 ± 0.100 0.103 ± 0.105 0.033 ± 0.019 0.030 ± 0.024 EP 0.131 ± 0.106 0.132 ± 0.111 0.033 ± 0.019 0.035 ± 0.022 ALEP 0.112 ± 0.095 0.103 ± 0.104 0.033 ± 0.019 0.030 ± 0.024 SFALEP 0.107 ± 0.086 0.098 ± 0.085 0.034 ± 0.019 0.031 ± 0.024 Calibration of models evaluated on GEFCom data with exogenous variables Forecast with RNN on Synthetic data Forecast with RNN on GEFCom data with exogenous variables Conclusions Successful application of the method proposed in [6] for probabilis- tic forecasting. A naive baseline proposed by us which helps with diagnosis and it is a good baseline for evaluation of models cali- bration. The proposed model, SFALEP, improved the results both from the perspective of performance, and calibration. RNN was more effective especially in the case of the real dataset with multi- ple variables involved in the forecasting. References [1] Christopher M. Bishop. Pattern Recognition and Machine Learning (Infor- mation Science and Statistics). Springer-Verlag, Berlin, Heidelberg, 2006. [2] Junyoung Chung, Yoshua Bengio, and et.al. Empirical evaluation of gated re- current neural networks on sequence modeling. CoRR, abs/1412.3555, 2014. [3] Yarin Gal and Zoubin Ghahramani. Dropout As a Bayesian Approxima- tion: Representing Model Uncertainty in Deep Learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML’16, pages 1050–1059. JMLR.org, 2016. [4] Sebastian Haglund and El Gaidi. Bounded Probabilistic Wind Power Fore- casting using Mixture Density Recurrent Neural Network. In Workshop on the Meaning of 42, 2018. [5] Tao Hong, Rob J. Hyndman, and et.al. Probabilistic energy forecasting: Global energy forecasting competition 2014 and beyond. International Jour- nal of Forecasting, 32(3):896 – 913, 2016. [6]Alex Kendall and Yarin Gal. What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? CoRR, abs/1703.0, 2017. [7] Juan.M Morales, Conejo, and et.al. Integrating Renewables in Electricity Markets. International Series in Operations Research & Management Sci- ence. Springer, 2011. [8] Leslie N. Smith. No more pesky learning rate guessing games. CoRR, abs/1506.01186, 2015. Acknowledgements I would like to gratefully acknowledge my master thesis supervisors Prof. Dr. Meelis Kull of the Insitute of Computer Science at University of Tartu, Erik Ylip of the Swedish Institute of Computer Science at Research Institutes of Sweden (RISE), and Sebastian Haglund El Gaidi at Greenlytics AB. I am also grateful for the encouraging advice and support from Sofiya Demchuck.
Transcript of with Monte-Carlo Dropout in Neural Networks · [7]Juan.M Morales, Conejo, and et.al. Integrating...
![Page 1: with Monte-Carlo Dropout in Neural Networks · [7]Juan.M Morales, Conejo, and et.al. Integrating Renewables in Electricity Markets. International Series in Operations Research & Management](https://reader035.fdocuments.net/reader035/viewer/2022071516/613835790ad5d20676491db6/html5/thumbnails/1.jpg)
Deep Probabilistic Forecastingwith Monte-Carlo Dropout in Neural Networks
Novin ShahroudiInstitute of Computer Science, University of TartuR [email protected] — � github.com/novinsh/master thesis — 7 @novinsh
Forecasting is an act of predicting the future that comes with a de-gree of uncertainty. That is why a probabilistic expression of thefuture suits its remote and uncertain nature better than other forms.Deep Learning is taking a prevalent role in today’s industries. It isbeing employed in more mission-critical tasks such as self-drivingcars as well as tasks that may not require a safety measure, yetrequire some measure of reliability such as in energy power fore-casting. Taking uncertainty into account in these models is a keyenabler to account for safety and reliability in these models. Thisposter demonstrates some of the experiments performed for mymaster thesis to perform probabilistic forecasting using a type ofVariational Bayesian Approximation for neural networks.
Reliable forecasting models play a crucial role in operations ofelectricity grid. Wind power is expected to provide at least 50% ofthe electricity production in Estonia and Sweden in the upcomingdecade. Therefore, this work was motivated to be applied on WindPower Forecasting.
Background
Time-series ForecastingIn the Time series forecasting, a history of data up to time is givenand a prediction should be made for the future for a given num-ber of steps. Data points are dependent on each other and may notbe identically distributed. Each step in the forecast referred to as alead time, and the whole forecast steps referred to as horizon. Equa-tion (1) denotes the observed values of a time series from the pastto the present and the future.
y0, y1, . . . , yt, yt+1, . . . , yt+h h ∈ N>0 (1)
Additional features could be incorporated in univariate time seriesforecasting that it is referred to as exogenous variables.
Bayesian Deep LearningA Bayesian neural network (BNN) [1] considers a distribution overits parameters as well as its outputs as opposed to a generic ArtificialNeural Network (ANN) which may be referred to as Non-bayesianNeural Network as well. Calculating the posterior of a Bayesianmodel is intractable for neural networks due to humongous param-eter space even with relatively small networks. In [3] authors in-troduce Monte-Carlo Dropout (MCDO) network as an approxima-tion of BNN that gives reasonably good results to approximate aBayesian Neural Network.
Probabilistic ForecastingProbabilistic or Dense Forecast is the most informative form offorecast that estimates a probability distribution rather a point fore-cast for each time step. It is common to demonstrate a probabilisticforecast using fan-charts, each fan representing a quantile of theprobability distribution as depicted in Fig. 5.
Evaluation metricsCRPS [7] evaluates quality of a probabilistic forecast represented bya predictive cumulative distribution function Ft(y) against the truevalue or observed probability passed through a heaviside functionH(y, y) as shown in Eq. (2). An important property of CRPS is thatit measures the accuracy, as well as the sharpness of the predictionswhich means calibratedness of the forecasts is also reflected in thescore. When CRPS normalized, it is denoted as NCRPS.
NCRPS(y) =1
n(T)∑t∈T
∫ ∞−∞
(Ft+h|t(y)−H(y − yt+h)
)2 (2)
Mean Squared Error (MSE) has the same measurement unit as ofthe squared of the series evaluated for. It is also equivalent to thevariance of the model output if the model is unbiased and it is tra-ditionally used for point estimate forecasts.
MSE(h) =1
n(T)∑t∈T
(yt+h|t − yt+h)2 (3)
DataFor the experiments with Synthetic data, the TimeSynthwas usedto generate and sample from a sinusoidal signal with frequency.
For the experiments with real data, wind power dataset fromGlobal Energy Forecasting Competition 2014 [5] was used. Setupof the data performed according to [4]. Two sets of experimentsconducted on this dataset, a univariate forecasting, and univariateforecasting with 4 wind speed features as other predictors.
Baseline ModelsPersistent or naive forecast is a well-known time-series forecastingbaseline. It uses the last observation as the forecast.
Methods
Probabilistic BaselineNaive method extended to provide probabilistic baseline where agiven quantile of the training data is being used as the forecast.
ModelsMethods explained in this section were mainly inspired by [6]. AMulti-layer Perceptron (MLP), and a Recurrent Neural Network(RNN) with GRU cells [2] as depicted in Fig. 1 were used. Bothmodels provided two outputs, one estimating the mean and the otherthe variance of the forecast. 100 units were used with a dropout rateof 0.3 for both models. For the RNN, recurrent dropout was used.The batch size for all experiments set to 32. All models were run for10 epochs with Adam optimizer and Cyclical Learning Rate Sched-uler [8]. Activation function for the first layer set to linear for theMLP and tanh for the RNN. Activation of the first dense output thatestimate the mean set to hard sigmoid, and the second dense outputthat estimates the logarithm of the variance set to linear.
Loss1
+
Total loss
Loss2
Dense 100
flatten
Dense 1
Input
Dropout
Dense 2
Input
Dense 1
GRU 100 dropout
Dense 1
+
Total loss
Loss1 Loss2Loss1
+
Total loss
Loss2
Dense 100
flatten
Dense 1
Input
Dropout
Dense 2
Input
Dense 1
GRU 100 dropout
Dense 1
+
Total loss
Loss1 Loss2
Architecture of MLP (left) and RNN (right)
Models were trained to forecast one-step ahead and to achievemulti-step ahead forecast roll-forward forecasting was used.
Uncertainty in the data, also referred to as Aleatoric uncertainty,is mainly caused by partial observability and stochasticity in theunderlying process generating the data, and/or errors in measure-ments. It is estimated by the second output of the models. A newloss as shown in Eq. 4 is then used. In this way, the network is en-couraged not only minimize the mean but also the variance of thepredictions.
L(θ) = 1
N
N∑i=1
1
σ(~xi)2|| ~yi − f (~xi) ||2 +logσ(~xi)2 (4)
Uncertainty in the model also referred to as Epistemic uncertainty.It can be caused by model misspecification (a wrong parametricfamily), error in the estimation of the model’s parameters whenmodeling, exposing the model to novel examples and any other sys-tematic error in the model. This can be estimated by Dropout at testtime. It produces different possible outputs that can be thought tobe produced by different sub-networks which would imply the vari-ance of the model. A neural network can also be thought of as anensemble method for reducing the variance of prediction by using anumber of unbiased models.
Scenario ForecastingThe roll-forward forecasting produces biased results as a result ofthe model’s ignorance caused by treating forecast values as new ob-servations. To tackle this issue scenario forecasting was used whichalso enabled the model to produce different forecast trajectories asanother informative form of forecasting. Since the forecasts wereprobabilistic, one could sample from the forecast distribution andthe sampled values could be used instead as the next value in theinput sequence forecasting for all possible alternative futures.
Demonstration of constructing trajectories with scenario forecasting. On the top,possible samples drew from the forecast distribution (denoted by red) in order toperform scenario forecasting on these samples (denoted by blue). On the bottom,generated trajectories as a result of scenario forecasting demonstrated. Scenariosreveal possible multimodality in the data. Each of these scenarios forecasts per-formed as part of a MCDO simulation and so were averaged over all to obtain thefinal result as shown Figure 5
ResultsAll results obtained by evaluation on 20 different splits of the val-idation set. Input size for the Synthetic data was 24 and of 4 forthe GEFCom’14 with forecast horizon of size 48. Results of 5variation of each model per each architecture are being reportedhere. 4 of which correspond to the Bayesian models and to a non-Bayesian model (old). The Bayesian models are denoted as AL forAleatoric, the EP for Epistemic, the ALEP for Aleatoric+Epistemic,and SFALEP for the Scenario Forecast+ALEP. Finally, the proba-bilistic baseline denoted by QNaive.
NCRSP results obtained on the GEFCom’14 Dataset
NCRPS Univariate with Exogenous VariablesArchitectures MLP GRU MLP GRUQNaive 0.211± 0.103 0.211± 0.103 0.209± 0.092 0.209± 0.092AL 0.215± 0.157 0.218± 0.155 0.167± 0.121 0.169± 0.150EP 0.247± 0.197 0.263± 0.186 0.196± 0.154 0.157± 0.141ALEP 0.208± 0.147 0.217± 0.150 0.166± 0.097 0.159± 0.137
SFALEP 0.203± 0.149 0.183± 0.097 0.173± 0.104 0.144± 0.099
Results obtained on the Synthetic Dataset
Metrics CRPS MSEArchitectures MLP GRU MLP GRU
old N/A N/A 0.033± 0.020 0.035± 0.021QNaive 0.163± 0.055 0.163± 0.055 0.082± 0.014 0.082± 0.014AL 0.116± 0.100 0.103± 0.105 0.033± 0.019 0.030± 0.024EP 0.131± 0.106 0.132± 0.111 0.033± 0.019 0.035± 0.022ALEP 0.112± 0.095 0.103± 0.104 0.033± 0.019 0.030± 0.024SFALEP 0.107± 0.086 0.098± 0.085 0.034± 0.019 0.031± 0.024
Calibration of models evaluated on GEFCom data with exogenous variables
Forecast with RNN on Synthetic data
Forecast with RNN on GEFCom data with exogenous variables
ConclusionsSuccessful application of the method proposed in [6] for probabilis-tic forecasting. A naive baseline proposed by us which helps withdiagnosis and it is a good baseline for evaluation of models cali-bration. The proposed model, SFALEP, improved the results bothfrom the perspective of performance, and calibration. RNN wasmore effective especially in the case of the real dataset with multi-ple variables involved in the forecasting.
References[1] Christopher M. Bishop. Pattern Recognition and Machine Learning (Infor-
mation Science and Statistics). Springer-Verlag, Berlin, Heidelberg, 2006.[2] Junyoung Chung, Yoshua Bengio, and et.al. Empirical evaluation of gated re-
current neural networks on sequence modeling. CoRR, abs/1412.3555, 2014.[3] Yarin Gal and Zoubin Ghahramani. Dropout As a Bayesian Approxima-
tion: Representing Model Uncertainty in Deep Learning. In Proceedings ofthe 33rd International Conference on International Conference on MachineLearning - Volume 48, ICML’16, pages 1050–1059. JMLR.org, 2016.
[4] Sebastian Haglund and El Gaidi. Bounded Probabilistic Wind Power Fore-casting using Mixture Density Recurrent Neural Network. In Workshop onthe Meaning of 42, 2018.
[5] Tao Hong, Rob J. Hyndman, and et.al. Probabilistic energy forecasting:Global energy forecasting competition 2014 and beyond. International Jour-nal of Forecasting, 32(3):896 – 913, 2016.
[6] Alex Kendall and Yarin Gal. What Uncertainties Do We Need in BayesianDeep Learning for Computer Vision? CoRR, abs/1703.0, 2017.
[7] Juan.M Morales, Conejo, and et.al. Integrating Renewables in ElectricityMarkets. International Series in Operations Research & Management Sci-ence. Springer, 2011.
[8] Leslie N. Smith. No more pesky learning rate guessing games. CoRR,abs/1506.01186, 2015.
AcknowledgementsI would like to gratefully acknowledge my master thesis supervisors Prof. Dr.Meelis Kull of the Insitute of Computer Science at University of Tartu, Erik Ylipof the Swedish Institute of Computer Science at Research Institutes of Sweden(RISE), and Sebastian Haglund El Gaidi at Greenlytics AB. I am also grateful forthe encouraging advice and support from Sofiya Demchuck.
