Forecasting Non-Stationary Time Series without Recurrent ...
Transcript of Forecasting Non-Stationary Time Series without Recurrent ...
![Page 1: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/1.jpg)
Forecasting Non-Stationary Time Series withoutRecurrent Connections
AP Engelbrecht
Department of Industrial Enigneering, andComputer Science Division
Stellenbosch UniversitySouth Africa
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 1 / 30
![Page 2: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/2.jpg)
Presentation Outline I
1 Introduction
2 The Time Series Used
3 Recurrent Neural Networks
4 Dynamic Optimization Problems
5 Particle Swarm Optimization
6 PSO Training of NNs
7 Empirical Analysis
8 Conclusions
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 2 / 30
![Page 3: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/3.jpg)
Introduction
The main goal of this study was to investigate if recurrent connectionsor time delays are necessary when training neural networks (NNs) fornon-stationary time series prediction using a dynamic particle swarmoptimization (PSO) algorithm
Consider training of the NN as a dynamic optimization problem, due tothe statistical properties of the time series changing over time
The quantum-inspired PSO (QSO) is a dynamic PSO with the ability totrack optima in changing landscapes
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 3 / 30
![Page 4: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/4.jpg)
The Time SeriesPlots
International Airline Passengers
(AIP)Australian Wine Sales (AWS)
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 4 / 30
![Page 5: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/5.jpg)
The Time SeriesPlots (cont)
US Accidental Death (USD) Sunspot Annual Measure (SAM)
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 5 / 30
![Page 6: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/6.jpg)
The Time SeriesPlots (cont)
Hourly Internet Traffic (HIT)Daily Minimum Temperature
(DMT)
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 6 / 30
![Page 7: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/7.jpg)
The Time SeriesPlots (cont)
Mackey Glass (MG) Logistic Map (LM)
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 7 / 30
![Page 8: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/8.jpg)
Feedforward Neural Networks
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 8 / 30
![Page 9: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/9.jpg)
Recurrent Neural Networks
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 9 / 30
![Page 10: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/10.jpg)
Dynamic Optimization Problems
Training of a NN is an optimization problem, with the objective to findbest values for weights and biases such that a given error function isminimized
Forecasting a non-stationary time series is a dynamic optimizationprocess, due to the statistical properties of the time series changingover time
Dynamic optimization problems:search landscape properties change over timeoptima change over time, in value and in positionnew optima may appearexisting optima may disappearchanges further characterized by change severity and changefrequency
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 10 / 30
![Page 11: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/11.jpg)
Dynamic Optimization Problems (cont)
Implications Optimization Algorithms:Need to adjust values assigned to decision variables in order totrack changing optima, without re-optimizingFor NN training, need to adapt weight and bias values to cope withconcept drift, without re-trainingShould have the ability to escape local minimaNeed to continually inject diversity into the search
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 11 / 30
![Page 12: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/12.jpg)
Particle Swarm OptimizationIntroduction
What is particle swarm optimization (PSO)?a simple, computationally efficient optimization methodpopulation-based, stochastic searchindividuals follow very simple behaviors:
emulate the success of neighboring individuals,but also bias towards own experience of success
emergent behavior: discovery of optimal regions within a highdimensional search space
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 12 / 30
![Page 13: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/13.jpg)
Particle Swarm OptimizationMain Components
What are the main components?a swarm of particleseach particle represents a candidate solutionelements of a particle represent parameters to be optimized
The search process:Position updates
xi(t + 1) = xi(t) + vi(t + 1), xij(0) ⇠ U(xmin,j , xmax ,j)
Velocity (step size)drives the optimization processreflects experiential knowledge of the particles andsocially exchanged information about promisingareas in the search space
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 13 / 30
![Page 14: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/14.jpg)
Particle Swarm OptimizationInertia Weight PSO
used either the star (gbest PSO) or social (lbest PSO) topologyvelocity update per dimension:
vij(t + 1) = wvij(t) + c1r1j(t)[yij(t)� xij(t)] + c2r2j(t)[yij(t)� xij(t)]
vij(0) = 0w is the inertia weightc1, c2 are positive acceleration coefficientsr1j(t), r2j(t) ⇠ U(0, 1)note that a random number is sampled for each dimension
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 14 / 30
![Page 15: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/15.jpg)
Particle Swarm OptimizationPSO Algorithm
Create and initialize an nx -dimensional swarm, S;repeat
for each particle i = 1, . . . ,S.ns do
if f (S.xi) < f (S.yi) then
S.yi = S.xi ;end
for each particle i with particle i in its neighborhood do
if f (S.yi) < f (S.yi) then
S.yi = S.yi ;end
end
end
for each particle i = 1, . . . ,S.ns do
update the velocity and position;end
until stopping condition is true;Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 15 / 30
![Page 16: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/16.jpg)
Particle Swarm OptimizationQuantum-Inspired PSO (QSO)
Developed to find and track an optimum in changing searchlandscapesBased on quantum model of an atom, where orbiting electrons arereplaced by a quantum cloud which is a probability distributiongoverning the position of each electronSwarm contains
neutral particles following standard PSO updatescharged, or quantum particles, randomly placed within amulti-dimensional sphere
xi(t + 1) =⇢
xi(t) + vi(t + 1) if Qi = 0By(rcloud ) if Qi 6= 0
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 16 / 30
![Page 17: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/17.jpg)
Particle Swarm OptimizationCooperative PSO
For large-scale optimization problems, a divide-and-conquer approachto address the curse of dimensionality:
Each particle is split into K separate parts of smaller dimensionEach part is then optimized using a separate sub-swarmIf K = nx , each dimension is optimized by a separate sub-swarm
Cooperative quantum PSO (CQSO) uses QSO in the sub-swarms
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 17 / 30
![Page 18: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/18.jpg)
PSO Training of NNs
When using PSO to train a NN:each particle represents the weights and biases of one NNobjective function is a cost function, e.g. SSEto prevent hidden unit saturation, use ReLUany activation function in the output units
For non-stationary time series prediction:Used cooperative PSO with QSO in sub-swarmsRNNs used modified hyperbolic tangent:f (net) = 1.7159 tanh(1
3net)
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 18 / 30
![Page 19: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/19.jpg)
Control Parameters
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 19 / 30
![Page 20: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/20.jpg)
Dynamic Scenarios
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 20 / 30
![Page 21: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/21.jpg)
Performance Measure
Used the collective mean error,
Fmean(t) =PT
t=1 F (t)T
where F (t) is the MSE at time t
Number of independent runs: 30
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 21 / 30
![Page 22: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/22.jpg)
ResultsMG (Mackey Glass)
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 22 / 30
![Page 23: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/23.jpg)
ResultsHIT (Hourly Internet Traffic)
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 23 / 30
![Page 24: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/24.jpg)
ResultsDMT (Daily Minimum Temperature)
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 24 / 30
![Page 25: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/25.jpg)
ResultsSAM (Sunspot Annual Measure)
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 25 / 30
![Page 26: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/26.jpg)
ResultsLM (Logistic Map)
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 26 / 30
![Page 27: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/27.jpg)
ResultsAWS (Australian Wine Sales)
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 27 / 30
![Page 28: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/28.jpg)
ResultsAIP (International Airline Passengers)
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 28 / 30
![Page 29: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/29.jpg)
ResultsUSD (US Accidental Death)
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 29 / 30
![Page 30: Forecasting Non-Stationary Time Series without Recurrent ...](https://reader031.fdocuments.net/reader031/viewer/2022013022/61d1affaab79c3134148ae07/html5/thumbnails/30.jpg)
Conclusions
The aim of the study was to investigate if recurrent connections ordelays are necessary if a dynamic PSO is used to train a NN for timeseries prediction
Main observation:A FFNN trained with the cooperative quantum PSO performedbetter than the RNNs used for most problems and scenariosWhere the CQSO FFNN algorithm did not perform best,differences in performance were not statistically significant
Future work will:expand the study to other variants of recurrent NNs, and moretime seriesdevelop dynamic architecture optimization approaches
Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 30 / 30