Optimization in Economics and Finance - Some Advances in Non-Linear
Non-linear optimization
description
Transcript of Non-linear optimization
![Page 1: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/1.jpg)
Non-linear optimization
An overview, problems and a guide
![Page 2: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/2.jpg)
w2
Optimization
)(min wEw
Unconstraint non-linear optimization
nE :
nw
E(w)
w1
![Page 3: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/3.jpg)
Classes of Methods
Linear optimization Constraint <-> unconstraint Gradient based 1st order, 2nd order Genetic Algorithms,
Evolutionary Strategies Stochastic methods
(Simulated Annealing, Tabu Search, …)
![Page 4: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/4.jpg)
Ellipsoid
![Page 5: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/5.jpg)
Rosenbrock-function
![Page 6: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/6.jpg)
Cross-Function
![Page 7: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/7.jpg)
Canyon-function
![Page 8: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/8.jpg)
Step-function
![Page 9: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/9.jpg)
Performance criteria
Number of function evaluations Number of gradient calculation Time Number of fails Number of method params. Sensitivity of method params. Accuracy
![Page 10: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/10.jpg)
Methods
Direct methods Successive variation Hooke-Jeeves
Gradient based methods Gradient decent Back-propagation Polak-Ribiere
Second order methods Newton-Raphson BFGS
![Page 11: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/11.jpg)
Successive Variation
![Page 12: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/12.jpg)
Successive Variation
![Page 13: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/13.jpg)
Successive Variation
![Page 14: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/14.jpg)
Successive Variation
![Page 15: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/15.jpg)
Hooke-Jeeves
![Page 16: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/16.jpg)
Hooke-Jeeves
![Page 17: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/17.jpg)
Hooke-Jeeves
![Page 18: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/18.jpg)
Gradient descent
)( )()()1( ttt wEww
![Page 19: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/19.jpg)
Gradient descent
![Page 20: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/20.jpg)
Gradient descent
![Page 21: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/21.jpg)
Gradient Decent
![Page 22: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/22.jpg)
Gradient descent
![Page 23: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/23.jpg)
Gradient descent
![Page 24: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/24.jpg)
Back-propagation
)1()()( )( ttt wwEw
)1()()1( ttt www
Gradient decent Momentum
![Page 25: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/25.jpg)
Back-propagation
![Page 26: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/26.jpg)
Back-propagationError E
Cycle
![Page 27: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/27.jpg)
Conjugated gradients
cbwwAwwQ TT )(
Qn property
cbA nnn ,,
![Page 28: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/28.jpg)
Beam search
))((min )()( tt wEwE
![Page 29: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/29.jpg)
Polak-Ribiere
)1()()()( )( tttt dwEd
)1()()()1( tttt dww
)()(
)()]()([)1()1(
)()1()()(
tTt
tTttt
wEwE
wEwEwE
Beam search
![Page 30: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/30.jpg)
Polak-Ribiere
![Page 31: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/31.jpg)
Newton-method
Q1 property
)()( )(1)(2)()1( tttt wEwEww
![Page 32: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/32.jpg)
BFGS
2)()(
)()()()(
)()(
)()()()()(
)]([
][][
][
][][tTt
TtttTt
tTt
TttTttt
w
wwv
w
vwwvB
)( )()()()()1( ttttt wEGww )()1()( ttt BGG
)1()()( ttt www
)()( )1()()( ttt wEwE)()1()()( tttt Gwv
![Page 33: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/33.jpg)
BFGS
![Page 34: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/34.jpg)
Comparison: Ellipsoid
0
5
10
15
20
25
30
SV HJ GD BP PR BFGS
Timen(E)n(grad E)
![Page 35: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/35.jpg)
Comparison: Cross-Function
0
50
100
150
200
250Timen(E)n(grad E)
![Page 36: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/36.jpg)
Comparison: Rosenbrock-Function
0
50
100
150
200
250
300
350Timen(E)n(grad E)
![Page 37: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/37.jpg)
Comparison: Canyon-Function
0200400600800
100012001400160018002000
Timen(E)n(grad E)
n(E)=8983
![Page 38: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/38.jpg)
Comparison: Step-Function
050
100150200250300350400450500
Timen(E)n(grad E)
n(E)=2487 n(E)=2448
![Page 39: Non-linear optimization](https://reader036.fdocuments.net/reader036/viewer/2022062309/568151a9550346895dbfd778/html5/thumbnails/39.jpg)
Decision tree
#minima
MC / SA GA / ES Multi-start differentiable
aligned? elliptic?
channels?
#parameters
Complexity
Know
ledge
NM / LBFGS
#parameters coordinate axis
HJ / ROS ROS SV
PR / LBFGS BFGS
QP / RPROP BP
onefewsomemany
yesno
yesno yes
fewmany no yes
fewmany
no yes
flatcurved along axes
G / PR/ BFGS