Introduction to unconstrained optimization - direct search ...
Transcript of Introduction to unconstrained optimization - direct search ...
Introduction to
unconstrained optimization
- direct search methods
Jussi Hakanen
Post-doctoral researcher [email protected]
spring 2014 TIES483 Nonlinear optimization
Structure of optimization methods
Typically โ Constraint handling
converts the problem to (a series of) unconstrained problems
โ In unconstrained optimization a search direction is determined at each iteration
โ The best solution in the search direction is found with line search
spring 2014 TIES483 Nonlinear optimization
Constraint handling
method
Unconstrained
optimization
Line
search
Group discussion
1. What kind of optimality conditions there exist for
unconstrained optimization (๐ฅ โ ๐ ๐)?
2. List methods for unconstrained optimization?
โ what are their general ideas?
Discuss in small groups (3-4) for 15-20 minutes
Each group has a secretary who writes down the
answers of the group
At the end, we summarize what each group found
spring 2014 TIES483 Nonlinear optimization
Reminder: gradient and hessian Definition: If function ๐: ๐ ๐ โ ๐ is differentiable, then the
gradient ๐ป๐(๐ฅ) consists of the partial derivatives ๐๐(๐ฅ)
๐๐ฅ๐ i.e.
๐ป๐ ๐ฅ =๐๐(๐ฅ)
๐๐ฅ1, โฆ ,
๐๐(๐ฅ)
๐๐ฅ๐
๐
Definition: If ๐ is twice differentiable, then the matrix
๐ป ๐ฅ =
๐2๐(๐ฅ)
๐๐ฅ1๐๐ฅ1โฏ
๐2๐(๐ฅ)
๐๐ฅ1๐๐ฅ๐
โฎ โฑ โฎ๐2๐(๐ฅ)
๐๐ฅ๐๐๐ฅ1โฏ
๐2๐(๐ฅ)
๐๐ฅ๐๐๐ฅ๐
is called the Hessian (matrix) of ๐ at ๐ฅ
Result: If ๐ is twice continuously differentiable, then ๐2๐(๐ฅ)
๐๐ฅ๐๐๐ฅ๐=
๐2๐(๐ฅ)
๐๐ฅ๐๐๐ฅ๐
spring 2014 TIES483 Nonlinear optimization
Reminder: Definite Matrices
Definition: A symmetric ๐ ร ๐ matrix ๐ป is positive semidefinite if โ ๐ฅ โ โ๐
๐ฅ๐๐ป๐ฅ โฅ 0.
Definition: A symmetric ๐ ร ๐ matrix ๐ป is positive definite if
๐ฅ๐๐ป๐ฅ > 0 โ 0 โ ๐ฅ โ โ๐
Note: If โฅ โ โค (> โ <), then ๐ป is negative semidefinite (definite). If ๐ป is neither positive nor negative semidefinite, then it is indefinite.
Result: Let โ โ ๐ โ ๐ ๐ be open convex set and ๐: ๐ โ ๐ twice differentiable in ๐. Function ๐ is convex if and only if ๐ป(๐ฅโ) is positive semidefinite for all ๐ฅโ โ ๐.
Unconstraint problem
min ๐ ๐ฅ , ๐ . ๐ก. ๐ฅ โ ๐ ๐
Necessary conditions: Let ๐ be twice differentiable in ๐ฅโ. If ๐ฅโ is a local minimizer, then
โ ๐ป๐ ๐ฅโ = 0 (that is, ๐ฅโ is a critical point of ๐) and
โ ๐ป ๐ฅโ is positive semidefinite.
Sufficient conditions: Let ๐ be twice differentiable in ๐ฅโ. If
โ ๐ป๐ ๐ฅโ = 0 and
โ ๐ป(๐ฅโ) is positive definite,
then ๐ฅโ is a strict local minimizer.
Result: Let ๐: ๐ ๐ โ ๐ is twice differentiable in ๐ฅโ. If ๐ป๐ ๐ฅโ = 0 and ๐ป(๐ฅโ) is indefinite, then ๐ฅโ is a saddle point.
Unconstraint problem
Adopted from Prof. L.T. Biegler (Carnegie Mellon
University)
Descent direction
Definition: Let ๐: ๐ ๐ โ ๐ . A vector ๐ โ ๐ ๐ is a descent
direction for ๐ in ๐ฅโ โ ๐ ๐ if โ ๐ฟ > 0 s.t.
๐ ๐ฅโ + ๐๐ < ๐(๐ฅโ) โ ๐ โ (0, ๐ฟ].
Result: Let ๐: ๐ ๐ โ ๐ be differentiable in ๐ฅโ. If โ๐ โ ๐ ๐
s.t. ๐ป๐ ๐ฅโ ๐๐ < 0 then ๐ is a descent direction for ๐ in
๐ฅโ.
spring 2014 TIES483 Nonlinear optimization
Model algorithm for unconstrained
minimization
Let ๐ฅโ be the current estimate for ๐ฅโ
1) [Test for convergence.] If conditions are satisfied, stop. The solution is ๐ฅโ.
2) [Compute a search direction.] Compute a non-zero vector ๐โ โ ๐ ๐ which is the search direction.
3) [Compute a step length.] Compute ๐ผโ > 0, the step length, for which it holds that ๐ ๐ฅโ + ๐ผโ๐โ < ๐(๐ฅโ).
4) [Update the estimate for minimum.] Set ๐ฅโ+1 = ๐ฅโ + ๐ผโ๐โ, โ = โ + 1 and go to step 1.
spring 2014 TIES483 Nonlinear optimization
From Gill et al., Practical Optimization, 1981, Academic Press
On convergence
Iterative method: a sequence {๐ฅโ} s.t. ๐ฅโ โ ๐ฅโ when โ โ โ
Definition: A method converges โ linearly if โ๐ผ โ [0,1) and ๐ โฅ 0 s.t. โ โ โฅ ๐
๐ฅโ+1 โ ๐ฅโ โค ๐ผ ๐ฅโ โ ๐ฅโ ,
โ superlinearly if โ๐ โฅ 0 and for some sequence ๐ผโ โ 0 it holds that โโ โฅ ๐
๐ฅโ+1 โ ๐ฅโ โค ๐ผโ ๐ฅโ โ ๐ฅโ ,
โ with degree ๐ if โ๐ผ โฅ 0, ๐ > 0 and ๐ โฅ 0 s.t. โโ โฅ ๐ ๐ฅโ+1 โ ๐ฅโ โค ๐ผ ๐ฅโ โ ๐ฅโ ๐
.
If ๐ = 2 (๐ = 3), the convergence is quadratic (cubic).
spring 2014 TIES483 Nonlinear optimization
Summary of group discussion for
methods
1. Newtonโs method 1. Utilizes tangent
2. Golden section method 1. For line search
3. Downhill Simplex
4. Cyclic coordinate method 1. One coordinate at a time
5. Polytopy search (Nelder-Mead) 1. Idea based on geometry
6. Gradient descent (steepest descent) 1. Based on gradient information
spring 2014 TIES483 Nonlinear optimization
Direct search methods
Univariate search, coordinate descent, cyclic
coordinate search
Hooke and Jeeves
Powellโs method
spring 2014 TIES483 Nonlinear optimization
Coordinate descent
spring 2014 TIES483 Nonlinear optimization
Fro
m M
iett
ine
n: N
on
line
ar
op
tim
iza
tio
n, 2
00
7 (
in F
inn
ish
)
๐ ๐ฅ = 2๐ฅ12 + 2๐ฅ1๐ฅ2 + ๐ฅ2
2 + ๐ฅ1 โ ๐ฅ2
Idea of pattern search
spring 2014 TIES483 Nonlinear optimization
Fro
m M
iett
ine
n: N
on
line
ar
op
tim
iza
tio
n, 2
00
7 (
in F
inn
ish
)
Hooke and Jeeves
spring 2014 TIES483 Nonlinear optimization
Fro
m M
iett
ine
n: N
on
line
ar
op
tim
iza
tio
n, 2
00
7 (
in F
inn
ish
)
๐ ๐ฅ = ๐ฅ1 โ 2 4 + x1 โ 2x22
Hooke and Jeeves with fixed step length
spring 2014 TIES483 Nonlinear optimization
Fro
m M
iett
ine
n: N
on
line
ar
op
tim
iza
tio
n, 2
00
7 (
in F
inn
ish
)
๐ ๐ฅ = ๐ฅ1 โ 2 4 + x1 โ 2x22
Powellโs method
Most efficient pattern search method
Differs from Hooke and Jeeves so that for
each pattern search step one of the coordinate
directions is replaced with previous pattern
search direction.
spring 2014 TIES483 Nonlinear optimization