Nonlinear Optimization
Claudia Schillings
HU Berlin - 14. October 2015
based on material by Michael Hintermuller, HU Berlin, Thomas Surowiec, HU Berlin
C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 1 / 12
Contents
Notions of solutions
Optimality conditions for unconstrained problems
Unconstrained optimization by descent methods with step sizestrategies
Convergence rates
Gradient based methods
Conjugate gradient method
Newton’s method
Quasi-Newton methods
Optimality conditions forconstrained problems
Algorithms forconstrained problems
C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 2 / 12
Contents
Notions of solutions
Optimality conditions for unconstrained problems
Unconstrained optimization by descent methods with step sizestrategies
Convergence rates
Gradient based methods
Conjugate gradient method
Newton’s method
Quasi-Newton methods
Optimality conditions forconstrained problems
Algorithms forconstrained problems
C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 2 / 12
Applications
Biological applications
Engineering systems
Environmental systems
Physical systems
...
Source: Chen et al.
...almost everywhere
C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 3 / 12
Applications
Biological applications
Engineering systems
Environmental systems
Physical systems
...
Copyright DLR.
...almost everywhere
C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 3 / 12
Applications
Biological applications
Engineering systems
Environmental systems
Physical systems
...
Source: Muggeridge et al.
...almost everywhere
C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 3 / 12
Applications
Biological applications
Engineering systems
Environmental systems
Physical systems
...
Copyright photonics.com.
...almost everywhere
C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 3 / 12
Applications
Biological applications
Engineering systems
Environmental systems
Physical systems
...
Copyright photonics.com.
...almost everywhere
C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 3 / 12
Organizational Stuff
Lectures:
Wednesdays 13-15 RUD25 1.013
Thursdays 13-15 RUD25 1.115
Exercise classes:
Thursdays 15-17 RUD25 2.006.
First problem sheet will be available tomorrow (15.10.15) for downloadon our course homepage.
The first exercise class will take place on Thursday, October 22nd.
Course homepage:
www.mathematik.hu-berlin.de/de/forschung/schillings-lehre/schillings-non-opt/
C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 4 / 12
Organizational Stuff
Assessment:
Final Exam
Problems to be solved on the board in the exercise classes.
Coding exercises (MATLAB).
Lecture notes:
Lecture notes will be provided as the module progresses.
Contact details:
Rudower Chaussee 25 , Room 2.408
Office hours: by appointment
www.mathematik.hu-berlin.de/de/personen/mitarb-vz/schillings-claudia
C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 4 / 12
Organizational Stuff
Literature:D. Bertsekas, Nonlinear Programming, Athena Scientific Publisher, Belmont,Massachusetts, 1995.
A. R. Conn, N. I. M. Gould, P. L. Toint, Trust-Region Methods, SIAM, Philadelphia, 2000.
J. E. Dennis, R. B. Schnabel, Numerical Methods for Unconstrained Optimization andNonlinear Equations, SIAM Philadelphia, 1996.
R. Fletcher, Practical Methods of Optimization, Wiley & Sons Publisher, New York, 1980.
C. Geiger, C. Kanzow, Numerische Verfahren zur Loesung unrestringierterOptimierungsaufgaben, Springer-Verlag, Berlin, 1999.
C. T. Kelley, Iterative Methods for Optimization, Frontiers in Applied Mathematics, SIAM,Philadelphia, 1999.
J. Nocedal and S. J. Wright, Numerical Optimization, Springer-Verlag, Berlin, 1999.
Prerequisites:
Linear Algebra, Analysis I + II, Analytic geometry
C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 4 / 12
Chapter 1: Introduction
Definition of a finite dimensional minimization problemLet X ⊂ Rn an arbitrary set and f : X → R a continuous function.The problem is to find an x∗ ∈ X such that
(1.1) f (x∗) ≤ f (x) for all x ∈ X .
Alternate formulations:
min f (x) s.t. x ∈ X ,
or
(1.2) minx∈X
f (x) .
C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 5 / 12
Chapter 1: Introduction
Definition of a finite dimensional minimization problemLet X ⊂ Rn an arbitrary set and f : X → R a continuous function.The problem is to find an x∗ ∈ X such that
(1.1) f (x∗) ≤ f (x) for all x ∈ X .
Alternate formulations:
min f (x) s.t. x ∈ X ,
or
(1.2) minx∈X
f (x) .
C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 5 / 12
Examples of Optimization ProblemsBasic IdeaGiven observations or measurements of a system of interest, how canwe determine certain intrinsic properties?
Undamped harmonic oscillatorLet M be a point of mass with mass m fixed to the end of avertical spring.
At equilibrium, M is located at the origin.
K is the restoring force which tries to replace M in itsequilibrium position.
For small (vertical) displacements y, the force K can bemodeled by Hooke’s law
K = −ky ,
where k denotes the (positive) spring constant.
http://people.seas.harvard.edu/jones/cscie129/nu lectures.
C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 6 / 12
Examples of Optimization Problems
Undamped harmonic oscillator
k denotes the unknown (positive) spring constant.
y(t) := position of M at time t.
Ignoring damping and friction, Newton’s law states:
(1.3) my = −ky ,
i.e. mass m times acceleration y equals the opposing force of the spring −ky.
(1.3) is called the undamped harmonic oscillator equation.
Usually, friction and damping forces behave proportionally to the velocity of M,i.e. −ry with fixed r > 0.
Together with (1.3), we obtain
my + ry + ky = 0 .
Setting c := r/m and k := k/m, we get
(1.4) y + cy + ky = 0 .
C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 7 / 12
Examples of Optimization ProblemsUndamped harmonic oscillator
Assume at time
(1.5) y(0) = y0 , y(0) = 0 .
Given endtime T > 0, we consider the initial boundary value problem (IVP) onthe interval [0, T].
The objective is to determine x = (c, k)> with the help of measurements.
For j = 1, . . . ,N, we are given measurements {yj}Nj=1 of the spring deviation at
time tj = (j− 1)T/(N − 1).
Let y(x; t) be the solution of the IVP for a given x. By solving the unconstrainedoptimization problem
(1.6) minx∈R2
f (x) :=12
N∑j=1
|y(x; tj)−yj|2 ,
we seek to determine the spring constant k and damping factor c.
Note that y(·; t) is differentiable w.r.t. x provided c2 − 4k 6= 0 .
(1.6) is a nonlinear least squares problem.
C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 8 / 12
Examples of Optimization Problems
Basic IdeaDeciding product capacity based on fixed and variable costs.
x output quantity.
Kv(x) variable costs, Kf (x) = c > 0 fixed costs.
K(x) := Kv(x) + Kf (x), x ∈ R total costs.
Normally one looks for an x∗, which minimizes total costs K(x), i.e.
(1.7) x∗ = argmin{Kv(x)+Kf (x) : x ∈ R} = argmin{Kv(x) : x ∈ R} .
In general, x∗ is not unique. We therefore write:
x∗ ∈ argmin{Kv(x) : x ∈ R} .
C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 9 / 12
Constrained OptimizationParticularly in the previous example, one often has constraints on x.
When X 6= Rn, we often have
X = X1 ∩ X2 ∩ X3
with sets
X1 = {x ∈ Rn : ci(x) = 0, i ∈ I1} ,X2 = {x ∈ Rn : ci(x) ≤ 0, i ∈ I2} ,X3 = {x ∈ Rn : xi ∈ Z, i ∈ I3} .
Ii ⊂ N, i = 1, 2, 3, are called (finite) index sets.
X1,X2,X3 are called equality, inequality and integer constraints.
X set of discrete points→ discrete or combinatorial optimization.
Otherwise, continuous optimization.
f , ci for any i is non differentiable→ nonsmooth optimization.
C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 10 / 12
Constrained OptimizationParticularly in the previous example, one often has constraints on x.
When X 6= Rn, we often have
X = X1 ∩ X2 ∩ X3
with sets
X1 = {x ∈ Rn : ci(x) = 0, i ∈ I1} ,X2 = {x ∈ Rn : ci(x) ≤ 0, i ∈ I2} ,X3 = {x ∈ Rn : xi ∈ Z, i ∈ I3} .
Ii ⊂ N, i = 1, 2, 3, are called (finite) index sets.
X1,X2,X3 are called equality, inequality and integer constraints.
X set of discrete points→ discrete or combinatorial optimization.
Otherwise, continuous optimization.
f , ci for any i is non differentiable→ nonsmooth optimization.
C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 10 / 12
Notions of Solutions
Definition 1.1Let f : X → R with X ⊂ Rn. The point x∗ ∈ X is called a
(i) (strict) global minimizer of f (on X), if
f (x∗) ≤ f (x) (f (x∗) < f (x)) ∀x ∈ X \ {x∗} .
The optimal objective value f (x∗) is called (strict) global minimum.
(ii) (strict) local minimizer of f (on X) if there exists a neighborhood of U of x∗ suchthat
f (x∗) ≤ f (x) (f (x∗) < f (x)) ∀x ∈ (X ∩ U) \ {x∗} .The optimal objective value f (x∗) is called (strict) local minimum.
C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 11 / 12
Stationary Points
Let X ⊂ Rn be an open set and f : X → R be a differentiable function. We denoteits gradient by
∇f (x) =(∂f∂x1
(x), . . . ,∂f∂xn
(x))>
.
If f : X → R is directionally differentiable, then its directional derivative at x ∈ X indirection d ∈ Rn is denoted by
f ′(x; d) := limα↓0
f (x + αd)− f (x)α
.
Definition 1.2Let X ⊂ Rn be an open set and f : X → R be a continuously differentiable function.The point x∗ ∈ X is called a stationary point of f , if
∇f (x∗) = 0
holds true.
C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 12 / 12
Top Related