Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear...

274
Constrained Nonlinear Optimization Algorithms Andreas Wchter Department of Industrial Engineering and Management Sciences Northwestern University [email protected] Institute for Mathematics and its Applications University of Minnesota August 4, 2016 Andreas Wchter Constrained Nonlinear Optimization Algorithms

Transcript of Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear...

Page 1: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Constrained Nonlinear Optimization Algorithms

Andreas Wächter

Department of Industrial Engineering and Management SciencesNorthwestern University

[email protected]

Institute for Mathematics and its ApplicationsUniversity of Minnesota

August 4, 2016

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 2: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Constrained Nonlinear Optimization Problems

minx∈Rn

f (x)

s.t. cE (x) = 0cI(x) ≤ 0

(NLP)f : Rn −→ R

cE : Rn −→ RmE

cI : Rn −→ RmI

I We assume that all functions are twice continuously differentiable.

I No is convexity required.I Most algorithms for NLP have

I theoretical convergence guarantee only to stationary points;I ingredients that steer towards local minimizers.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 3: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Constrained Nonlinear Optimization Problems

minx∈Rn

f (x)

s.t. cE (x) = 0cI(x) ≤ 0

(NLP)f : Rn −→ R

cE : Rn −→ RmE

cI : Rn −→ RmI

I We assume that all functions are twice continuously differentiable.I No is convexity required.

I Most algorithms for NLP haveI theoretical convergence guarantee only to stationary points;I ingredients that steer towards local minimizers.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 4: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Constrained Nonlinear Optimization Problems

minx∈Rn

f (x)

s.t. cE (x) = 0cI(x) ≤ 0

(NLP)f : Rn −→ R

cE : Rn −→ RmE

cI : Rn −→ RmI

I We assume that all functions are twice continuously differentiable.I No is convexity required.I Most algorithms for NLP have

I theoretical convergence guarantee only to stationary points;

I ingredients that steer towards local minimizers.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 5: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Constrained Nonlinear Optimization Problems

minx∈Rn

f (x)

s.t. cE (x) = 0cI(x) ≤ 0

(NLP)f : Rn −→ R

cE : Rn −→ RmE

cI : Rn −→ RmI

I We assume that all functions are twice continuously differentiable.I No is convexity required.I Most algorithms for NLP have

I theoretical convergence guarantee only to stationary points;I ingredients that steer towards local minimizers.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 6: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Table of Contents

Applications

Equality-Constrained Quadratic Programming

Active-Set Quadratic Programming Solvers

SQP for Equality-Constrained NLPs

SQP for Inequality-Constrained NLPs

Interior Point Methods

Software

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 7: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Table of Contents

Applications

Equality-Constrained Quadratic Programming

Active-Set Quadratic Programming Solvers

SQP for Equality-Constrained NLPs

SQP for Inequality-Constrained NLPs

Interior Point Methods

Software

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 8: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Design and Operation of Chemical Plant

I Minimize “Costs − Profit”I Variables: Physical quantitiesI Constraints: physical relationships

(conservation laws; themodyn. rel.)I Limits (physical and operational)I < 105 variables; few degrees of freedom

Constraint Jacobian

∇c(x)T =

is sparse and structured

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 9: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Design and Operation of Chemical Plant

I Minimize “Costs − Profit”I Variables: Physical quantitiesI Constraints: physical relationships

(conservation laws; themodyn. rel.)I Limits (physical and operational)I < 105 variables; few degrees of freedom

Constraint Jacobian

∇c(x)T =

is sparse and structured

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 10: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Design and Operation of Chemical Plant

I Minimize “Costs − Profit”I Variables: Physical quantitiesI Constraints: physical relationships

(conservation laws; themodyn. rel.)I Limits (physical and operational)I < 105 variables; few degrees of freedom

Constraint Jacobian

∇c(x)T =

is sparse and structured

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 11: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Design Under Uncertainty

∇c(x)T =

I Scenario parameters: F lin, x lin,T l

env , plenv , . . . (given)(defining scenarios l = 1, . . . , L)

I Design variables: Vreact ,Ddist , htank , . . .I Control variables: Ql

heat , r lrefl , v lvalve , . . .I State variables: x lstr ,F l

str , Lltank ,T l , pl , . . .

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 12: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Design Under Uncertainty

∇c(x)T =

I Scenario parameters: F lin, x lin,T l

env , plenv , . . . (given)(defining scenarios l = 1, . . . , L)

I Design variables: Vreact ,Ddist , htank , . . .I Control variables: Ql

heat , r lrefl , v lvalve , . . .I State variables: x lstr ,F l

str , Lltank ,T l , pl , . . .

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 13: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Optimal Control / Dynamic Optimization

minz,y ,u,p

f (z(tf ), y(tf ), u(tf ), p)

s.t. F (z(t), z(t), y(t), u(t), p) = 0G(z(t), y(t), u(t), p) = 0z(0) = zinitbound constraints

u : [0, tf ]→ Rnu control variablesz : [0, tf ]→ Rnz differentiable state variablesy : [0, tf ]→ Rny algebraic state variablesp ∈ Rnp time-independent parameterszinit initial conditionstf final time

I Large-scale NLPs arise from discretization.Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 14: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Circuit Tuning

I Model consists of network of gates.I Gate delays computed by simulation (expensive, noisy).I Model has many variables (up to 1, 000, 000).I Implemented in IBM’s circuit tuning tool EinsTuner.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 15: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Hyperthermia Treatment Planning

minT (x),u

12

∫Ω

(T (x)− Ttarget(x))2 dx

s.t. −∆T (x)− w(T (x)− Tblood) = u∗M(x)u in Ω

∇T (x) · n = Texterior − T (x) on ∂Ω

T (x) ≤ Tmax in Ω \ ΩTumor

I Heat tumors with microwaves (support chemo- and radio-therapy).I Model is a PDE with

I controls: Application u of microwave antennas.I states: Temperature T (x) defined over domain Ω.

I Finite-dimensional problem obtained by discretization.I e.g., finite differences, finite elements

I Resulting NLP is usually very large.Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 16: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Table of Contents

Applications

Equality-Constrained Quadratic Programming

Active-Set Quadratic Programming Solvers

SQP for Equality-Constrained NLPs

SQP for Inequality-Constrained NLPs

Interior Point Methods

Software

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 17: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Quadratic Programming

minx∈Rn

12x

TQx + gT x

s.t. AEx + bE = 0AIx + bI ≤ 0

(QP)Q ∈ Rn×n symmetric

AE ∈ RmE×n bE ∈ RmE

AI ∈ RmI×n bI ∈ RmI

I Many applications (e.g., portfolio optimization, optimal control).I Important building block for methods for general NLP.I Algorithms:

I Active-set methodsI Interior-point methods

I Let’s first consider equality-constrained case.I Assume: all rows of AE are linearly independent.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 18: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Quadratic Programming

minx∈Rn

12x

TQx + gT x

s.t. AEx + bE = 0AIx + bI ≤ 0

(QP)Q ∈ Rn×n symmetric

AE ∈ RmE×n bE ∈ RmE

AI ∈ RmI×n bI ∈ RmI

I Many applications (e.g., portfolio optimization, optimal control).

I Important building block for methods for general NLP.I Algorithms:

I Active-set methodsI Interior-point methods

I Let’s first consider equality-constrained case.I Assume: all rows of AE are linearly independent.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 19: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Quadratic Programming

minx∈Rn

12x

TQx + gT x

s.t. AEx + bE = 0AIx + bI ≤ 0

(QP)Q ∈ Rn×n symmetric

AE ∈ RmE×n bE ∈ RmE

AI ∈ RmI×n bI ∈ RmI

I Many applications (e.g., portfolio optimization, optimal control).I Important building block for methods for general NLP.

I Algorithms:I Active-set methodsI Interior-point methods

I Let’s first consider equality-constrained case.I Assume: all rows of AE are linearly independent.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 20: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Quadratic Programming

minx∈Rn

12x

TQx + gT x

s.t. AEx + bE = 0AIx + bI ≤ 0

(QP)Q ∈ Rn×n symmetric

AE ∈ RmE×n bE ∈ RmE

AI ∈ RmI×n bI ∈ RmI

I Many applications (e.g., portfolio optimization, optimal control).I Important building block for methods for general NLP.I Algorithms:

I Active-set methodsI Interior-point methods

I Let’s first consider equality-constrained case.I Assume: all rows of AE are linearly independent.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 21: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Quadratic Programming

minx∈Rn

12x

TQx + gT x

s.t. AEx + bE = 0AIx + bI ≤ 0

(QP)Q ∈ Rn×n symmetric

AE ∈ RmE×n bE ∈ RmE

AI ∈ RmI×n bI ∈ RmI

I Many applications (e.g., portfolio optimization, optimal control).I Important building block for methods for general NLP.I Algorithms:

I Active-set methodsI Interior-point methods

I Let’s first consider equality-constrained case.I Assume: all rows of AE are linearly independent.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 22: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Equality-Constrained QP

minx∈Rn

12x

TQx + gT x

s.t. Ax + b = 0(EQP)

First-order optimality conditions:

Qx + g + ATλ = 0Ax + b = 0

Find stationary point (x∗, λ∗) by solving the linear system[Q AT

A 0

](x∗λ∗

)= −

(gb

).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 23: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Equality-Constrained QP

minx∈Rn

12x

TQx + gT x

s.t. Ax + b = 0(EQP)

First-order optimality conditions:

Qx + g + ATλ = 0Ax + b = 0

Find stationary point (x∗, λ∗) by solving the linear system[Q AT

A 0

](x∗λ∗

)= −

(gb

).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 24: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Equality-Constrained QP

minx∈Rn

12x

TQx + gT x

s.t. Ax + b = 0(EQP)

First-order optimality conditions:

Qx + g + ATλ = 0Ax + b = 0

Find stationary point (x∗, λ∗) by solving the linear system[Q AT

A 0

](x∗λ∗

)= −

(gb

).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 25: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

KKT System of QP

[Q AT

A 0

](x∗λ∗

)= −

(gb

)

I When is (x∗, λ∗) indeed a solution of (EQP)?

I Recall the second-order optimality conditions:I Let the columns of Z ∈ Rn×(n−m) be a basis of the null-space of A,

so AZ = 0 (“null-space matrix”).I Then x∗ is a strict local minimizer of (EQP) if ZTQZ 0.

I On the other hand:I If ZTQZ has negative eigenvalue, then (EQP) is unbounded below.

I There are different ways to solve the KKT systemI Best choice depends on particular problem

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 26: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

KKT System of QP

[Q AT

A 0

](x∗λ∗

)= −

(gb

)

I When is (x∗, λ∗) indeed a solution of (EQP)?I Recall the second-order optimality conditions:

I Let the columns of Z ∈ Rn×(n−m) be a basis of the null-space of A,so AZ = 0 (“null-space matrix”).

I Then x∗ is a strict local minimizer of (EQP) if ZTQZ 0.

I On the other hand:I If ZTQZ has negative eigenvalue, then (EQP) is unbounded below.

I There are different ways to solve the KKT systemI Best choice depends on particular problem

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 27: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

KKT System of QP

[Q AT

A 0

](x∗λ∗

)= −

(gb

)

I When is (x∗, λ∗) indeed a solution of (EQP)?I Recall the second-order optimality conditions:

I Let the columns of Z ∈ Rn×(n−m) be a basis of the null-space of A,so AZ = 0 (“null-space matrix”).

I Then x∗ is a strict local minimizer of (EQP) if ZTQZ 0.

I On the other hand:I If ZTQZ has negative eigenvalue, then (EQP) is unbounded below.

I There are different ways to solve the KKT systemI Best choice depends on particular problem

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 28: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

KKT System of QP

[Q AT

A 0

](x∗λ∗

)= −

(gb

)

I When is (x∗, λ∗) indeed a solution of (EQP)?I Recall the second-order optimality conditions:

I Let the columns of Z ∈ Rn×(n−m) be a basis of the null-space of A,so AZ = 0 (“null-space matrix”).

I Then x∗ is a strict local minimizer of (EQP) if ZTQZ 0.

I On the other hand:I If ZTQZ has negative eigenvalue, then (EQP) is unbounded below.

I There are different ways to solve the KKT systemI Best choice depends on particular problem

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 29: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Direct Solution of the KKT System[Q AT

A 0

]︸ ︷︷ ︸

=:K

(x∗λ∗

)= −

(gb

)

I How can we verify that x∗ is local minimizer without computing Z?

DefinitionLet n+, n−, n0 be the number of positive, negative, and zero eigenvaluesof a matrix M. Then In(M) = (n+, n−, n0) is the inertia of M.

TheoremSuppose that A has full rank. Then: In(K ) = In(ZTQZ ) + (m,m, 0).

CorollaryIf In(K ) = (n,m, 0), then x∗ is the unique global minimizer.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 30: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Direct Solution of the KKT System[Q AT

A 0

]︸ ︷︷ ︸

=:K

(x∗λ∗

)= −

(gb

)

I How can we verify that x∗ is local minimizer without computing Z?

DefinitionLet n+, n−, n0 be the number of positive, negative, and zero eigenvaluesof a matrix M. Then In(M) = (n+, n−, n0) is the inertia of M.

TheoremSuppose that A has full rank. Then: In(K ) = In(ZTQZ ) + (m,m, 0).

CorollaryIf In(K ) = (n,m, 0), then x∗ is the unique global minimizer.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 31: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Direct Solution of the KKT System[Q AT

A 0

]︸ ︷︷ ︸

=:K

(x∗λ∗

)= −

(gb

)

I How can we verify that x∗ is local minimizer without computing Z?

DefinitionLet n+, n−, n0 be the number of positive, negative, and zero eigenvaluesof a matrix M. Then In(M) = (n+, n−, n0) is the inertia of M.

TheoremSuppose that A has full rank. Then: In(K ) = In(ZTQZ ) + (m,m, 0).

CorollaryIf In(K ) = (n,m, 0), then x∗ is the unique global minimizer.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 32: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Direct Solution of the KKT System[Q AT

A 0

]︸ ︷︷ ︸

=:K

(x∗λ∗

)= −

(gb

)

I How can we verify that x∗ is local minimizer without computing Z?

DefinitionLet n+, n−, n0 be the number of positive, negative, and zero eigenvaluesof a matrix M. Then In(M) = (n+, n−, n0) is the inertia of M.

TheoremSuppose that A has full rank. Then: In(K ) = In(ZTQZ ) + (m,m, 0).

CorollaryIf In(K ) = (n,m, 0), then x∗ is the unique global minimizer.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 33: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Computing the Inertia

[Q AT

A 0

]︸ ︷︷ ︸

=:K

(x∗λ∗

)= −

(gb

)

I Symmetric indefinite factorization PKPT = LBLTI P: permutation matrixI L: unit lower triangular matrixI B: block diagonal matrix with 1× 1 and 2× 2 diagonal blocks

I Can be computed efficiently, exploits sparsity.I Obtain inertia simply from counting eigenvalues of the blocks in B.I Used also to solve the linear system.I Will be important later when we need to “convexify” QPs

(Q ← Q + γI).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 34: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Computing the Inertia

[Q AT

A 0

]︸ ︷︷ ︸

=:K

(x∗λ∗

)= −

(gb

)

I Symmetric indefinite factorization PKPT = LBLTI P: permutation matrixI L: unit lower triangular matrixI B: block diagonal matrix with 1× 1 and 2× 2 diagonal blocks

I Can be computed efficiently, exploits sparsity.

I Obtain inertia simply from counting eigenvalues of the blocks in B.I Used also to solve the linear system.I Will be important later when we need to “convexify” QPs

(Q ← Q + γI).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 35: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Computing the Inertia

[Q AT

A 0

]︸ ︷︷ ︸

=:K

(x∗λ∗

)= −

(gb

)

I Symmetric indefinite factorization PKPT = LBLTI P: permutation matrixI L: unit lower triangular matrixI B: block diagonal matrix with 1× 1 and 2× 2 diagonal blocks

I Can be computed efficiently, exploits sparsity.I Obtain inertia simply from counting eigenvalues of the blocks in B.

I Used also to solve the linear system.I Will be important later when we need to “convexify” QPs

(Q ← Q + γI).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 36: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Computing the Inertia

[Q AT

A 0

]︸ ︷︷ ︸

=:K

(x∗λ∗

)= −

(gb

)

I Symmetric indefinite factorization PKPT = LBLTI P: permutation matrixI L: unit lower triangular matrixI B: block diagonal matrix with 1× 1 and 2× 2 diagonal blocks

I Can be computed efficiently, exploits sparsity.I Obtain inertia simply from counting eigenvalues of the blocks in B.I Used also to solve the linear system.

I Will be important later when we need to “convexify” QPs(Q ← Q + γI).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 37: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Computing the Inertia

[Q AT

A 0

]︸ ︷︷ ︸

=:K

(x∗λ∗

)= −

(gb

)

I Symmetric indefinite factorization PKPT = LBLTI P: permutation matrixI L: unit lower triangular matrixI B: block diagonal matrix with 1× 1 and 2× 2 diagonal blocks

I Can be computed efficiently, exploits sparsity.I Obtain inertia simply from counting eigenvalues of the blocks in B.I Used also to solve the linear system.I Will be important later when we need to “convexify” QPs

(Q ← Q + γI).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 38: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Schur-Complement Method

Qx + g + ATλ = 0Ax + b = 0

I Assume, Q is positive definite. Then AQ−1AT is nonsingular.

I Pre-multiply first equation by AQ−1.I Then solve

[AQ−1AT ]λ∗ = b − AQ−1g

Qx = −g − ATλ∗

I Requirements:I Solutions with Q can be done efficientlyI Need to compute [AQ−1AT ] and solve linear system with itI Works best if m is small

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 39: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Schur-Complement Method

Qx + g + ATλ = 0Ax + b = 0

I Assume, Q is positive definite. Then AQ−1AT is nonsingular.I Pre-multiply first equation by AQ−1.

I Then solve[AQ−1AT ]λ∗ = b − AQ−1g

Qx = −g − ATλ∗

I Requirements:I Solutions with Q can be done efficientlyI Need to compute [AQ−1AT ] and solve linear system with itI Works best if m is small

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 40: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Schur-Complement Method

Qx + g + ATλ = 0Ax + b = 0

I Assume, Q is positive definite. Then AQ−1AT is nonsingular.I Pre-multiply first equation by AQ−1.I Then solve

[AQ−1AT ]λ∗ = b − AQ−1g

Qx = −g − ATλ∗

I Requirements:I Solutions with Q can be done efficientlyI Need to compute [AQ−1AT ] and solve linear system with itI Works best if m is small

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 41: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Schur-Complement Method

Qx + g + ATλ = 0Ax + b = 0

I Assume, Q is positive definite. Then AQ−1AT is nonsingular.I Pre-multiply first equation by AQ−1.I Then solve

[AQ−1AT ]λ∗ = b − AQ−1gQx = −g − ATλ∗

I Requirements:I Solutions with Q can be done efficientlyI Need to compute [AQ−1AT ] and solve linear system with itI Works best if m is small

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 42: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Schur-Complement Method

Qx + g + ATλ = 0Ax + b = 0

I Assume, Q is positive definite. Then AQ−1AT is nonsingular.I Pre-multiply first equation by AQ−1.I Then solve

[AQ−1AT ]λ∗ = b − AQ−1gQx = −g − ATλ∗

I Requirements:I Solutions with Q can be done efficiently

I Need to compute [AQ−1AT ] and solve linear system with itI Works best if m is small

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 43: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Schur-Complement Method

Qx + g + ATλ = 0Ax + b = 0

I Assume, Q is positive definite. Then AQ−1AT is nonsingular.I Pre-multiply first equation by AQ−1.I Then solve

[AQ−1AT ]λ∗ = b − AQ−1gQx = −g − ATλ∗

I Requirements:I Solutions with Q can be done efficientlyI Need to compute [AQ−1AT ] and solve linear system with it

I Works best if m is small

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 44: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Schur-Complement Method

Qx + g + ATλ = 0Ax + b = 0

I Assume, Q is positive definite. Then AQ−1AT is nonsingular.I Pre-multiply first equation by AQ−1.I Then solve

[AQ−1AT ]λ∗ = b − AQ−1gQx = −g − ATλ∗

I Requirements:I Solutions with Q can be done efficientlyI Need to compute [AQ−1AT ] and solve linear system with itI Works best if m is small

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 45: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Step Decomposition

0

I Decompose x∗ = xY + xZ into two steps:I “range-space step” xY : step into constraintsI “null-space step” xZ : optimize within null space

I xY = YpY and xZ = ZpZI where [Y Z ] is basis of Rn and Z is null space matrix for A.

I Decomposition depends on choice of Y and Z

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 46: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Step Decomposition

0

x∗

I Decompose x∗ = xY + xZ into two steps:I “range-space step” xY : step into constraintsI “null-space step” xZ : optimize within null space

I xY = YpY and xZ = ZpZI where [Y Z ] is basis of Rn and Z is null space matrix for A.

I Decomposition depends on choice of Y and Z

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 47: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Step Decomposition

0

x∗

xY

xZ

I Decompose x∗ = xY + xZ into two steps:I “range-space step” xY : step into constraintsI “null-space step” xZ : optimize within null space

I xY = YpY and xZ = ZpZI where [Y Z ] is basis of Rn and Z is null space matrix for A.

I Decomposition depends on choice of Y and Z

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 48: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Step Decomposition

0

x∗

xY

xZ

I Decompose x∗ = xY + xZ into two steps:I “range-space step” xY : step into constraintsI “null-space step” xZ : optimize within null space

I xY = YpY and xZ = ZpZI where [Y Z ] is basis of Rn and Z is null space matrix for A.

I Decomposition depends on choice of Y and Z

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 49: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Step Decomposition

0

x∗

xY

xZ

I Decompose x∗ = xY + xZ into two steps:I “range-space step” xY : step into constraintsI “null-space step” xZ : optimize within null space

I xY = YpY and xZ = ZpZI where [Y Z ] is basis of Rn and Z is null space matrix for A.

I Decomposition depends on choice of Y and ZAndreas Wächter Constrained Nonlinear Optimization Algorithms

Page 50: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Step Decomposition

Qx + g + ATλ = 0Ax + b = 0

0

x∗

xY

xZ

I xY = YpY is a step into the constraints:

0 = Ax + b = AYpY + AZpZ + b

=⇒ pY = −[AY ]−1bI xZ = ZpZ optimizes in the null space

pZ = −[ZTQZ ]−1ZT (g + QYpY )

I Solves minpZ 12p

TZ [ZTQZ ]pZ + (g + QYpY )TZpZ (“reduced QP”)

I λ = −[AY ]−TY (Qx∗ + g)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 51: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Step Decomposition

Qx + g + ATλ = 0Ax + b = 0

0

x∗

xY

xZ

I xY = YpY is a step into the constraints:

0 = Ax + b = AYpY + AZpZ + b =⇒ pY = −[AY ]−1b

I xZ = ZpZ optimizes in the null space

pZ = −[ZTQZ ]−1ZT (g + QYpY )

I Solves minpZ 12p

TZ [ZTQZ ]pZ + (g + QYpY )TZpZ (“reduced QP”)

I λ = −[AY ]−TY (Qx∗ + g)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 52: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Step Decomposition

Qx + g + ATλ = 0Ax + b = 0

0

x∗

xY

xZ

I xY = YpY is a step into the constraints:

0 = Ax + b = AYpY + AZpZ + b =⇒ pY = −[AY ]−1bI xZ = ZpZ optimizes in the null space

pZ = −[ZTQZ ]−1ZT (g + QYpY )

I Solves minpZ 12p

TZ [ZTQZ ]pZ + (g + QYpY )TZpZ (“reduced QP”)

I λ = −[AY ]−TY (Qx∗ + g)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 53: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Step Decomposition

Qx + g + ATλ = 0Ax + b = 0

0

x∗

xY

xZ

I xY = YpY is a step into the constraints:

0 = Ax + b = AYpY + AZpZ + b =⇒ pY = −[AY ]−1bI xZ = ZpZ optimizes in the null space

pZ = −[ZTQZ ]−1ZT (g + QYpY )

I Solves minpZ 12p

TZ [ZTQZ ]pZ + (g + QYpY )TZpZ (“reduced QP”)

I λ = −[AY ]−TY (Qx∗ + g)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 54: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Step Decomposition

Qx + g + ATλ = 0Ax + b = 0

0

x∗

xY

xZ

I xY = YpY is a step into the constraints:

0 = Ax + b = AYpY + AZpZ + b =⇒ pY = −[AY ]−1bI xZ = ZpZ optimizes in the null space

pZ = −[ZTQZ ]−1ZT (g + QYpY )

I Solves minpZ 12p

TZ [ZTQZ ]pZ + (g + QYpY )TZpZ (“reduced QP”)

I λ = −[AY ]−TY (Qx∗ + g)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 55: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Example: PDE-Constrained Optimization

minT ,u

12

∫(T (z)− T (z))2dz +

α

2 ‖u‖2

s.t. −∆T (z) =nu∑i=1

ki(z)ui on Ω

T (z) = b(z) on ∂Ω

I Given the (independent) control variable u:I (Dependent) state T is solution of PDEI Can use well-established solution techniques

I We have only nu degrees of freedom

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 56: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Example: PDE-Constrained Optimization

minT ,u

12

∫(T (z)− T (z))2dz +

α

2 ‖u‖2

s.t. −∆T (z) =nu∑i=1

ki(z)ui on Ω

T (z) = b(z) on ∂Ω

I Given the (independent) control variable u:I (Dependent) state T is solution of PDEI Can use well-established solution techniques

I We have only nu degrees of freedom

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 57: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Example: PDE-Constrained Optimization

minT ,u

12

∫(T (z)− T (z))2dz +

α

2 ‖u‖2

s.t. −∆T (z) =nu∑i=1

ki(z)ui on Ω

T (z) = b(z) on ∂Ω

I Given the (independent) control variable u:I (Dependent) state T is solution of PDEI Can use well-established solution techniques

I We have only nu degrees of freedom

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 58: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Discretized PDE-Constrained Problem

minT ,u

12

∫(T (z)− T (z))2dz +

α

2 ‖u‖2

s.t. −∆T (z) =nu∑i=1

ki(z)ui on Ω

T (z) = b(z) on ∂Ω

mint,u

n∑i=1

(ti − ti)2 +nu∑i=1

u2i

s.t. D t + Ku + b = 0

I Discretized state variables t ∈ RN

I Discretized non-singular(!) differential operator D ∈ RN×N

I Given controls u, the state variables can be computed from

t = −D−1(Ku + b).

I We could just eliminate t and solve lower-dimensional problem in u

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 59: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Discretized PDE-Constrained Problem

minT ,u

12

∫(T (z)− T (z))2dz +

α

2 ‖u‖2

s.t. −∆T (z) =nu∑i=1

ki(z)ui on Ω

T (z) = b(z) on ∂Ω

mint,u

n∑i=1

(ti − ti)2 +nu∑i=1

u2i

s.t. D t + Ku + b = 0

I Discretized state variables t ∈ RN

I Discretized non-singular(!) differential operator D ∈ RN×N

I Given controls u, the state variables can be computed from

t = −D−1(Ku + b).

I We could just eliminate t and solve lower-dimensional problem in u

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 60: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Discretized PDE-Constrained Problem

minT ,u

12

∫(T (z)− T (z))2dz +

α

2 ‖u‖2

s.t. −∆T (z) =nu∑i=1

ki(z)ui on Ω

T (z) = b(z) on ∂Ω

mint,u

n∑i=1

(ti − ti)2 +nu∑i=1

u2i

s.t. D t + Ku + b = 0

I Discretized state variables t ∈ RN

I Discretized non-singular(!) differential operator D ∈ RN×N

I Given controls u, the state variables can be computed from

t = −D−1(Ku + b).

I We could just eliminate t and solve lower-dimensional problem in u

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 61: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Generalization: Elimination of Variables

minx∈Rn

12(xTB xTN )Q

(xBxN

)+ gTB xB + gTN xN

s.t. BxB + NxN + b = 0

Y =

[I0

]

Z =

[−B−1N

I

]

pY = −[AY ]−1d = −B−1bpZ = −[ZTQZ ]−1ZT (g + QYpY )

λ = −[AY ]−TY (Qx∗ + g) = −B−TY (Qx∗ + g)

I Can use existing implementations of operator B−1:I Compute Z and pZ (assuming N has few columns).I Compute λ∗ (assuming that we have implementation for B−T ).

I Tailored implementation for “simulation” often already exist.I Exploit problem structure!

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 62: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Generalization: Elimination of Variables

minx∈Rn

12(xTB xTN )Q

(xBxN

)+ gTB xB + gTN xN

s.t. BxB + NxN + b = 0

Y =

[I0

]

Z =

[−B−1N

I

]

pY = −[AY ]−1d = −B−1bpZ = −[ZTQZ ]−1ZT (g + QYpY )

λ = −[AY ]−TY (Qx∗ + g) = −B−TY (Qx∗ + g)

I Can use existing implementations of operator B−1:I Compute Z and pZ (assuming N has few columns).I Compute λ∗ (assuming that we have implementation for B−T ).

I Tailored implementation for “simulation” often already exist.I Exploit problem structure!

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 63: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Generalization: Elimination of Variables

minx∈Rn

12(xTB xTN )Q

(xBxN

)+ gTB xB + gTN xN

s.t. BxB + NxN + b = 0

Y =

[I0

]

Z =

[−B−1N

I

]

pY = −[AY ]−1d = −B−1bpZ = −[ZTQZ ]−1ZT (g + QYpY )

λ = −[AY ]−TY (Qx∗ + g) = −B−TY (Qx∗ + g)

I Can use existing implementations of operator B−1:I Compute Z and pZ (assuming N has few columns).I Compute λ∗ (assuming that we have implementation for B−T ).

I Tailored implementation for “simulation” often already exist.

I Exploit problem structure!

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 64: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Generalization: Elimination of Variables

minx∈Rn

12(xTB xTN )Q

(xBxN

)+ gTB xB + gTN xN

s.t. BxB + NxN + b = 0

Y =

[I0

]

Z =

[−B−1N

I

]

pY = −[AY ]−1d = −B−1bpZ = −[ZTQZ ]−1ZT (g + QYpY )

λ = −[AY ]−TY (Qx∗ + g) = −B−TY (Qx∗ + g)

I Can use existing implementations of operator B−1:I Compute Z and pZ (assuming N has few columns).I Compute λ∗ (assuming that we have implementation for B−T ).

I Tailored implementation for “simulation” often already exist.I Exploit problem structure!

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 65: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Solution of EQP Summary

I Direct method:I Factorize KKT matrixI If LTBL factorization is used, we can determine if x∗ is indeed a

minimizerI Easy general purpose option

I Schur-complement MethodI Requires that Q is positive definite and easy to solve (e.g., diagonal)I Number of constraints m should not be large

I Null-space methodI Step decomposition into range-space step and null-space stepI Permits exploitation of constraint matrix structureI Number of degrees of freedom (n −m) should not be large

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 66: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Solution of EQP Summary

I Direct method:I Factorize KKT matrixI If LTBL factorization is used, we can determine if x∗ is indeed a

minimizerI Easy general purpose option

I Schur-complement MethodI Requires that Q is positive definite and easy to solve (e.g., diagonal)I Number of constraints m should not be large

I Null-space methodI Step decomposition into range-space step and null-space stepI Permits exploitation of constraint matrix structureI Number of degrees of freedom (n −m) should not be large

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 67: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Solution of EQP Summary

I Direct method:I Factorize KKT matrixI If LTBL factorization is used, we can determine if x∗ is indeed a

minimizerI Easy general purpose option

I Schur-complement MethodI Requires that Q is positive definite and easy to solve (e.g., diagonal)I Number of constraints m should not be large

I Null-space methodI Step decomposition into range-space step and null-space stepI Permits exploitation of constraint matrix structureI Number of degrees of freedom (n −m) should not be large

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 68: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Table of Contents

Applications

Equality-Constrained Quadratic Programming

Active-Set Quadratic Programming Solvers

SQP for Equality-Constrained NLPs

SQP for Inequality-Constrained NLPs

Interior Point Methods

Software

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 69: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Inequality-Constrained QPs

minx∈Rn

12x

TQx + gT x

s.t. aTi x + bi = 0 for i ∈ EaTi x + bi ≤ 0 for i ∈ I

Qx + g +∑

i∈E∪Iaiλi = 0

aTi x + bi = 0 for i ∈ EaTi x + bi ≤ 0 for i ∈ I

λi ≥ 0 for i ∈ I(aTi x + bi)λi = 0 for i ∈ I

I Assume:I Q is positive definite;I aii∈E are linearly independent.

I Difficulty: Decide, which inequality constraints are active.I We know how to solve equality-constrained QPs.

I Can we use that here?

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 70: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Inequality-Constrained QPs

minx∈Rn

12x

TQx + gT x

s.t. aTi x + bi = 0 for i ∈ EaTi x + bi ≤ 0 for i ∈ I

Qx + g +∑

i∈E∪Iaiλi = 0

aTi x + bi = 0 for i ∈ EaTi x + bi ≤ 0 for i ∈ I

λi ≥ 0 for i ∈ I(aTi x + bi)λi = 0 for i ∈ I

I Assume:I Q is positive definite;I aii∈E are linearly independent.

I Difficulty: Decide, which inequality constraints are active.I We know how to solve equality-constrained QPs.

I Can we use that here?

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 71: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Inequality-Constrained QPs

minx∈Rn

12x

TQx + gT x

s.t. aTi x + bi = 0 for i ∈ EaTi x + bi ≤ 0 for i ∈ I

Qx + g +∑

i∈E∪Iaiλi = 0

aTi x + bi = 0 for i ∈ EaTi x + bi ≤ 0 for i ∈ I

λi ≥ 0 for i ∈ I(aTi x + bi)λi = 0 for i ∈ I

I Assume:I Q is positive definite;I aii∈E are linearly independent.

I Difficulty: Decide, which inequality constraints are active.I We know how to solve equality-constrained QPs.

I Can we use that here?

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 72: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Inequality-Constrained QPs

minx∈Rn

12x

TQx + gT x

s.t. aTi x + bi = 0 for i ∈ EaTi x + bi ≤ 0 for i ∈ I

Qx + g +∑

i∈E∪Iaiλi = 0

aTi x + bi = 0 for i ∈ EaTi x + bi ≤ 0 for i ∈ I

λi ≥ 0 for i ∈ I(aTi x + bi)λi = 0 for i ∈ I

I Assume:I Q is positive definite;I aii∈E are linearly independent.

I Difficulty: Decide, which inequality constraints are active.

I We know how to solve equality-constrained QPs.I Can we use that here?

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 73: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Inequality-Constrained QPs

minx∈Rn

12x

TQx + gT x

s.t. aTi x + bi = 0 for i ∈ EaTi x + bi ≤ 0 for i ∈ I

Qx + g +∑

i∈E∪Iaiλi = 0

aTi x + bi = 0 for i ∈ EaTi x + bi ≤ 0 for i ∈ I

λi ≥ 0 for i ∈ I(aTi x + bi)λi = 0 for i ∈ I

I Assume:I Q is positive definite;I aii∈E are linearly independent.

I Difficulty: Decide, which inequality constraints are active.I We know how to solve equality-constrained QPs.

I Can we use that here?

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 74: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Working SetChoose working set W ⊆ I and solve

minx∈Rn

12x

TQx + gT x

s.t. aTi x + bi = 0 for i ∈ EaTi x + bi = 0 for i ∈ W

Qx + g +∑

i∈E∪Waiλi = 0

aTi x + bi = 0 for i ∈ EaTi x + bi = 0 for i ∈ W

Set missing multipliers λi = 0 for i ∈ I \W and verify

aTi x + bi?≤ 0 for i ∈ I \W

λi?≥ 0 for i ∈ I

I If satisfied, (x , λ) is the (unique) optimal solutionI Otherwise, let’s try a different working set

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 75: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Working SetChoose working set W ⊆ I and solve

minx∈Rn

12x

TQx + gT x

s.t. aTi x + bi = 0 for i ∈ EaTi x + bi = 0 for i ∈ W

Qx + g +∑

i∈E∪Waiλi = 0

aTi x + bi = 0 for i ∈ EaTi x + bi = 0 for i ∈ W

Set missing multipliers λi = 0 for i ∈ I \W and verify

aTi x + bi?≤ 0 for i ∈ I \W

λi?≥ 0 for i ∈ I

I If satisfied, (x , λ) is the (unique) optimal solutionI Otherwise, let’s try a different working set

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 76: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Working SetChoose working set W ⊆ I and solve

minx∈Rn

12x

TQx + gT x

s.t. aTi x + bi = 0 for i ∈ EaTi x + bi = 0 for i ∈ W

Qx + g +∑

i∈E∪Waiλi = 0

aTi x + bi = 0 for i ∈ EaTi x + bi = 0 for i ∈ W

Set missing multipliers λi = 0 for i ∈ I \W and verify

aTi x + bi?≤ 0 for i ∈ I \W

λi?≥ 0 for i ∈ I

I If satisfied, (x , λ) is the (unique) optimal solutionI Otherwise, let’s try a different working set

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 77: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Working SetChoose working set W ⊆ I and solve

minx∈Rn

12x

TQx + gT x

s.t. aTi x + bi = 0 for i ∈ EaTi x + bi = 0 for i ∈ W

Qx + g +∑

i∈E∪Waiλi = 0

aTi x + bi = 0 for i ∈ EaTi x + bi = 0 for i ∈ W

Set missing multipliers λi = 0 for i ∈ I \W and verify

aTi x + bi?≤ 0 for i ∈ I \W

λi?≥ 0 for i ∈ I

I If satisfied, (x , λ) is the (unique) optimal solution

I Otherwise, let’s try a different working set

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 78: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Working SetChoose working set W ⊆ I and solve

minx∈Rn

12x

TQx + gT x

s.t. aTi x + bi = 0 for i ∈ EaTi x + bi = 0 for i ∈ W

Qx + g +∑

i∈E∪Waiλi = 0

aTi x + bi = 0 for i ∈ EaTi x + bi = 0 for i ∈ W

Set missing multipliers λi = 0 for i ∈ I \W and verify

aTi x + bi?≤ 0 for i ∈ I \W

λi?≥ 0 for i ∈ I

I If satisfied, (x , λ) is the (unique) optimal solutionI Otherwise, let’s try a different working set

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 79: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Example QP

(3)

(1)(2)

(4)

(5)

min (x1 − 1)2 + (x2 − 2.5)2

s.t. − x1 + 2x2 − 2 ≤ 0 (1) −x1 ≤ 0 (4)x1 + 2x2 − 6 ≤ 0 (2) −x2 ≤ 0 (5)x1 − 2x2 − 2 ≤ 0 (3)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 80: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 1

(3)

(1)(2)

(4)

(5)

W = 3, 5

x = (0, 2)

xEQP = (0, 2)

λ3 = −2λ5 = −1

Initialization:Choose feasible starting iterate x

Choose working set W ⊆ I withI i ∈ W =⇒ aTi x + bi = 0I aii∈E∪W are linear independent

(Algorithm will maintain these properties)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 81: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 1

(3)

(1)(2)

(4)

(5)

W = 3, 5x = (0, 2)

xEQP = (0, 2)

λ3 = −2λ5 = −1

Initialization:Choose feasible starting iterate xChoose working set W ⊆ I with

I i ∈ W =⇒ aTi x + bi = 0I aii∈E∪W are linear independent

(Algorithm will maintain these properties)Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 82: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 1

(3)

(1)(2)

(4)

(5)

W = 3, 5x = (0, 2)

xEQP = (0, 2)

λ3 = −2λ5 = −1

Solve (EQP)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 83: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 1

(3)

(1)(2)

(4)

(5)

W = 3, 5x = (0, 2)

xEQP = (0, 2)

λ3 = −2λ5 = −1

Status: Current iterate is optimal for (EQP).

Release Constraint:I Pick constraint i with λi < 0.

I Remove i from working set:W ←W \ 3 = 5

I Keep iterate x = (0, 2).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 84: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 1

(3)

(1)(2)

(4)

(5)

W = 3, 5x = (0, 2)

xEQP = (0, 2)

λ3 = −2λ5 = −1

Status: Current iterate is optimal for (EQP).

Release Constraint:I Pick constraint i with λi < 0.

I Remove i from working set:W ←W \ 3 = 5

I Keep iterate x = (0, 2).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 85: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 1

(3)

(1)(2)

(4)

(5)

W = 3, 5x = (0, 2)

xEQP = (0, 2)

λ3 = −2λ5 = −1

Status: Current iterate is optimal for (EQP).

Release Constraint:I Pick constraint i with λi < 0.I Remove i from working set:

W ←W \ 3 = 5

I Keep iterate x = (0, 2).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 86: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 1

(3)

(1)(2)

(4)

(5)

W = 3, 5x = (0, 2)

xEQP = (0, 2)

λ3 = −2λ5 = −1

Status: Current iterate is optimal for (EQP).

Release Constraint:I Pick constraint i with λi < 0.I Remove i from working set:

W ←W \ 3 = 5I Keep iterate x = (0, 2).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 87: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 2

(3)

(1)(2)

(4)

(5)

W = 5x = (2, 0)

xEQP = (1, 0)

λ5 = −5

Solve (EQP)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 88: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 2

(3)

(1)(2)

(4)

(5)

W = 5x = (2, 0)

xEQP = (1, 0)

λ5 = −5

Status: Current iterate is not optimal for (EQP).

Take step (xEQP is feasible):I Update iterate x ← xEQP

I Keep W

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 89: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 2

(3)

(1)(2)

(4)

(5)

W = 5x = (2, 0)

xEQP = (1, 0)

λ5 = −5

Status: Current iterate is not optimal for (EQP).

Take step (xEQP is feasible):I Update iterate x ← xEQP

I Keep W

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 90: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 2

(3)

(1)(2)

(4)

(5)

W = 5x = (2, 0)

xEQP = (1, 0)

λ5 = −5

Status: Current iterate is not optimal for (EQP).

Take step (xEQP is feasible):I Update iterate x ← xEQP

I Keep W

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 91: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 3

(3)

(1)(2)

(4)

(5)

W = 5x = (1, 0)

xEQP = (1, 0)

λ5 = −5

Solve (EQP)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 92: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 3

(3)

(1)(2)

(4)

(5)

W = 5x = (1, 0)

xEQP = (1, 0)

λ5 = −5

Status: Current iterate is optimal for (EQP)

Release Constraint:I Pick constraint i with λi < 0.I Remove i from working set:

W ←W \ 5 = ∅I Keep iterate x = (1, 0).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 93: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 3

(3)

(1)(2)

(4)

(5)

W = 5x = (1, 0)

xEQP = (1, 0)

λ5 = −5

Status: Current iterate is optimal for (EQP)

Release Constraint:I Pick constraint i with λi < 0.

I Remove i from working set:W ←W \ 5 = ∅

I Keep iterate x = (1, 0).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 94: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 3

(3)

(1)(2)

(4)

(5)

W = 5x = (1, 0)

xEQP = (1, 0)

λ5 = −5

Status: Current iterate is optimal for (EQP)

Release Constraint:I Pick constraint i with λi < 0.I Remove i from working set:

W ←W \ 5 = ∅

I Keep iterate x = (1, 0).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 95: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 3

(3)

(1)(2)

(4)

(5)

W = 5x = (1, 0)

xEQP = (1, 0)

λ5 = −5

Status: Current iterate is optimal for (EQP)

Release Constraint:I Pick constraint i with λi < 0.I Remove i from working set:

W ←W \ 5 = ∅I Keep iterate x = (1, 0).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 96: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 4

(3)

(1)(2)

(4)

(5)

W = ∅x = (1, 0)

xEQP = (1, 2.5)

Solve (EQP)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 97: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 4

(3)

(1)(2)

(4)

(5)

W = ∅x = (1, 0)

xEQP = (1, 2.5)

Status: Current iterate not optimal for (EQP)

Take step (xEQP not feasible):

I Largest α ∈ [0, 1]: x + α(xEQP − x) feasibleI Update iterate x ← x + α(xEQP − x)I Update W ←W ∪ i = 1

I where constraint i = 1 is “blocking”

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 98: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 4

(3)

(1)(2)

(4)

(5)

W = ∅x = (1, 0)

xEQP = (1, 2.5)

Status: Current iterate not optimal for (EQP)

Take step (xEQP not feasible):

I Largest α ∈ [0, 1]: x + α(xEQP − x) feasibleI Update iterate x ← x + α(xEQP − x)I Update W ←W ∪ i = 1

I where constraint i = 1 is “blocking”

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 99: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 4

(3)

(1)(2)

(4)

(5)

W = ∅x = (1, 0)

xEQP = (1, 2.5)

Status: Current iterate not optimal for (EQP)

Take step (xEQP not feasible):I Largest α ∈ [0, 1]: x + α(xEQP − x) feasible

I Update iterate x ← x + α(xEQP − x)I Update W ←W ∪ i = 1

I where constraint i = 1 is “blocking”

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 100: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 4

(3)

(1)(2)

(4)

(5)

W = ∅x = (1, 0)

xEQP = (1, 2.5)

Status: Current iterate not optimal for (EQP)

Take step (xEQP not feasible):I Largest α ∈ [0, 1]: x + α(xEQP − x) feasibleI Update iterate x ← x + α(xEQP − x)

I Update W ←W ∪ i = 1I where constraint i = 1 is “blocking”

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 101: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 4

(3)

(1)(2)

(4)

(5)

W = ∅x = (1, 0)

xEQP = (1, 2.5)

Status: Current iterate not optimal for (EQP)

Take step (xEQP not feasible):I Largest α ∈ [0, 1]: x + α(xEQP − x) feasibleI Update iterate x ← x + α(xEQP − x)I Update W ←W ∪ i = 1

I where constraint i = 1 is “blocking”Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 102: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 5

(3)

(1)(2)

(4)

(5)

W = 1x = (1, 1.5)

xEQP = (1.4, 1.7)

λ1 = 0.8

Solve (EQP)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 103: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 5

(3)

(1)(2)

(4)

(5)

W = 1x = (1, 1.5)

xEQP = (1.4, 1.7)

λ1 = 0.8

Status: Current iterate is not optimal for (EQP).

Take step (xEQP feasible):I Update iterate x ← xEQP.I Keep W.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 104: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 5

(3)

(1)(2)

(4)

(5)

W = 1x = (1, 1.5)

xEQP = (1.4, 1.7)

λ1 = 0.8

Status: Current iterate is not optimal for (EQP).

Take step (xEQP feasible):I Update iterate x ← xEQP.I Keep W.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 105: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 6

(3)

(1)(2)

(4)

(5)

W = 1x = (1.4, 1.7)

xEQP = (1.4, 1.7)

λ1 = 0.8

Solve (EQP)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 106: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 6

(3)

(1)(2)

(4)

(5)

W = 1x = (1.4, 1.7)

xEQP = (1.4, 1.7)

λ1 = 0.8

Status: Current iterate is optimal for (EQP)

I λi ≥ 0 for all i ∈ W.

Declare Optimality!

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 107: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 6

(3)

(1)(2)

(4)

(5)

W = 1x = (1.4, 1.7)

xEQP = (1.4, 1.7)

λ1 = 0.8

Status: Current iterate is optimal for (EQP)

I λi ≥ 0 for all i ∈ W.

Declare Optimality!

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 108: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Solver Iteration 6

(3)

(1)(2)

(4)

(5)

W = 1x = (1.4, 1.7)

xEQP = (1.4, 1.7)

λ1 = 0.8

Status: Current iterate is optimal for (EQP)

I λi ≥ 0 for all i ∈ W.

Declare Optimality!

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 109: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Method

1. Select feasible x and W ⊆ I ∩A(x).

2. Solve (EQP) to get xEQP and λEQP.3. If x = xEQP:

I If λEQP ≥ 0: Done!I Otherwise, select λEQP

i < 0 and set W ←W \ i.4. If x 6= xEQP:

I Compute step p = xEQP − x .I Compute α = argmaxα ∈ [0, 1] : x + αp is feasible.I If α < 1, pick i ∈ I \W with aTi p > 0 and aTi (x + αp) + bi = 0, and

set W ←W ∪ i.I Update x ← x + αp.

5. Go to step 2.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 110: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Method

1. Select feasible x and W ⊆ I ∩A(x).2. Solve (EQP) to get xEQP and λEQP.

3. If x = xEQP:I If λEQP ≥ 0: Done!I Otherwise, select λEQP

i < 0 and set W ←W \ i.4. If x 6= xEQP:

I Compute step p = xEQP − x .I Compute α = argmaxα ∈ [0, 1] : x + αp is feasible.I If α < 1, pick i ∈ I \W with aTi p > 0 and aTi (x + αp) + bi = 0, and

set W ←W ∪ i.I Update x ← x + αp.

5. Go to step 2.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 111: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Method

1. Select feasible x and W ⊆ I ∩A(x).2. Solve (EQP) to get xEQP and λEQP.3. If x = xEQP:

I If λEQP ≥ 0: Done!I Otherwise, select λEQP

i < 0 and set W ←W \ i.4. If x 6= xEQP:

I Compute step p = xEQP − x .I Compute α = argmaxα ∈ [0, 1] : x + αp is feasible.I If α < 1, pick i ∈ I \W with aTi p > 0 and aTi (x + αp) + bi = 0, and

set W ←W ∪ i.I Update x ← x + αp.

5. Go to step 2.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 112: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Method

1. Select feasible x and W ⊆ I ∩A(x).2. Solve (EQP) to get xEQP and λEQP.3. If x = xEQP:

I If λEQP ≥ 0: Done!

I Otherwise, select λEQPi < 0 and set W ←W \ i.

4. If x 6= xEQP:I Compute step p = xEQP − x .I Compute α = argmaxα ∈ [0, 1] : x + αp is feasible.I If α < 1, pick i ∈ I \W with aTi p > 0 and aTi (x + αp) + bi = 0, and

set W ←W ∪ i.I Update x ← x + αp.

5. Go to step 2.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 113: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Method

1. Select feasible x and W ⊆ I ∩A(x).2. Solve (EQP) to get xEQP and λEQP.3. If x = xEQP:

I If λEQP ≥ 0: Done!I Otherwise, select λEQP

i < 0 and set W ←W \ i.

4. If x 6= xEQP:I Compute step p = xEQP − x .I Compute α = argmaxα ∈ [0, 1] : x + αp is feasible.I If α < 1, pick i ∈ I \W with aTi p > 0 and aTi (x + αp) + bi = 0, and

set W ←W ∪ i.I Update x ← x + αp.

5. Go to step 2.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 114: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Method

1. Select feasible x and W ⊆ I ∩A(x).2. Solve (EQP) to get xEQP and λEQP.3. If x = xEQP:

I If λEQP ≥ 0: Done!I Otherwise, select λEQP

i < 0 and set W ←W \ i.4. If x 6= xEQP:

I Compute step p = xEQP − x .I Compute α = argmaxα ∈ [0, 1] : x + αp is feasible.I If α < 1, pick i ∈ I \W with aTi p > 0 and aTi (x + αp) + bi = 0, and

set W ←W ∪ i.I Update x ← x + αp.

5. Go to step 2.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 115: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Method

1. Select feasible x and W ⊆ I ∩A(x).2. Solve (EQP) to get xEQP and λEQP.3. If x = xEQP:

I If λEQP ≥ 0: Done!I Otherwise, select λEQP

i < 0 and set W ←W \ i.4. If x 6= xEQP:

I Compute step p = xEQP − x .

I Compute α = argmaxα ∈ [0, 1] : x + αp is feasible.I If α < 1, pick i ∈ I \W with aTi p > 0 and aTi (x + αp) + bi = 0, and

set W ←W ∪ i.I Update x ← x + αp.

5. Go to step 2.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 116: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Method

1. Select feasible x and W ⊆ I ∩A(x).2. Solve (EQP) to get xEQP and λEQP.3. If x = xEQP:

I If λEQP ≥ 0: Done!I Otherwise, select λEQP

i < 0 and set W ←W \ i.4. If x 6= xEQP:

I Compute step p = xEQP − x .I Compute α = argmaxα ∈ [0, 1] : x + αp is feasible.

I If α < 1, pick i ∈ I \W with aTi p > 0 and aTi (x + αp) + bi = 0, andset W ←W ∪ i.

I Update x ← x + αp.5. Go to step 2.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 117: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Method

1. Select feasible x and W ⊆ I ∩A(x).2. Solve (EQP) to get xEQP and λEQP.3. If x = xEQP:

I If λEQP ≥ 0: Done!I Otherwise, select λEQP

i < 0 and set W ←W \ i.4. If x 6= xEQP:

I Compute step p = xEQP − x .I Compute α = argmaxα ∈ [0, 1] : x + αp is feasible.I If α < 1, pick i ∈ I \W with aTi p > 0 and aTi (x + αp) + bi = 0, and

set W ←W ∪ i.

I Update x ← x + αp.5. Go to step 2.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 118: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal Active-Set QP Method

1. Select feasible x and W ⊆ I ∩A(x).2. Solve (EQP) to get xEQP and λEQP.3. If x = xEQP:

I If λEQP ≥ 0: Done!I Otherwise, select λEQP

i < 0 and set W ←W \ i.4. If x 6= xEQP:

I Compute step p = xEQP − x .I Compute α = argmaxα ∈ [0, 1] : x + αp is feasible.I If α < 1, pick i ∈ I \W with aTi p > 0 and aTi (x + αp) + bi = 0, and

set W ←W ∪ i.I Update x ← x + αp.

5. Go to step 2.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 119: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Active-Set QP AlgorithmsI Primal active-set method:

I Keeps all iterates feasible.I Changes W by at most one constraint per iteration.I aii∈E∪W remain linearly independent.

I ConvergenceI Finite convergence:

I Finitely many options for W.I Objective decreases with every step; as long as α > 0!

I Special handling of degeneracy necessary (α = 0 steps)I Efficient solution of (EQP)

I Update the factorization when W changes.I Complexity

I Fast convergence if good estimate of optimal working set is given.I Worst case exponential complexity.I Alternative: Interior-point QP solvers (polynomial complexity).

I There are variants that allow Q to be indefinite.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 120: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Active-Set QP AlgorithmsI Primal active-set method:

I Keeps all iterates feasible.I Changes W by at most one constraint per iteration.I aii∈E∪W remain linearly independent.

I ConvergenceI Finite convergence:

I Finitely many options for W.I Objective decreases with every step;

as long as α > 0!I Special handling of degeneracy necessary (α = 0 steps)

I Efficient solution of (EQP)I Update the factorization when W changes.

I ComplexityI Fast convergence if good estimate of optimal working set is given.I Worst case exponential complexity.I Alternative: Interior-point QP solvers (polynomial complexity).

I There are variants that allow Q to be indefinite.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 121: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Active-Set QP AlgorithmsI Primal active-set method:

I Keeps all iterates feasible.I Changes W by at most one constraint per iteration.I aii∈E∪W remain linearly independent.

I ConvergenceI Finite convergence:

I Finitely many options for W.I Objective decreases with every step; as long as α > 0!

I Special handling of degeneracy necessary (α = 0 steps)

I Efficient solution of (EQP)I Update the factorization when W changes.

I ComplexityI Fast convergence if good estimate of optimal working set is given.I Worst case exponential complexity.I Alternative: Interior-point QP solvers (polynomial complexity).

I There are variants that allow Q to be indefinite.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 122: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Active-Set QP AlgorithmsI Primal active-set method:

I Keeps all iterates feasible.I Changes W by at most one constraint per iteration.I aii∈E∪W remain linearly independent.

I ConvergenceI Finite convergence:

I Finitely many options for W.I Objective decreases with every step; as long as α > 0!

I Special handling of degeneracy necessary (α = 0 steps)I Efficient solution of (EQP)

I Update the factorization when W changes.

I ComplexityI Fast convergence if good estimate of optimal working set is given.I Worst case exponential complexity.I Alternative: Interior-point QP solvers (polynomial complexity).

I There are variants that allow Q to be indefinite.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 123: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Active-Set QP AlgorithmsI Primal active-set method:

I Keeps all iterates feasible.I Changes W by at most one constraint per iteration.I aii∈E∪W remain linearly independent.

I ConvergenceI Finite convergence:

I Finitely many options for W.I Objective decreases with every step; as long as α > 0!

I Special handling of degeneracy necessary (α = 0 steps)I Efficient solution of (EQP)

I Update the factorization when W changes.I Complexity

I Fast convergence if good estimate of optimal working set is given.I Worst case exponential complexity.I Alternative: Interior-point QP solvers (polynomial complexity).

I There are variants that allow Q to be indefinite.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 124: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Active-Set QP AlgorithmsI Primal active-set method:

I Keeps all iterates feasible.I Changes W by at most one constraint per iteration.I aii∈E∪W remain linearly independent.

I ConvergenceI Finite convergence:

I Finitely many options for W.I Objective decreases with every step; as long as α > 0!

I Special handling of degeneracy necessary (α = 0 steps)I Efficient solution of (EQP)

I Update the factorization when W changes.I Complexity

I Fast convergence if good estimate of optimal working set is given.I Worst case exponential complexity.I Alternative: Interior-point QP solvers (polynomial complexity).

I There are variants that allow Q to be indefinite.Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 125: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Table of Contents

Applications

Equality-Constrained Quadratic Programming

Active-Set Quadratic Programming Solvers

SQP for Equality-Constrained NLPs

SQP for Inequality-Constrained NLPs

Interior Point Methods

Software

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 126: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Equality-Constrained Nonlinear Problems

minx∈Rn

f (x)

s.t. c(x) = 0

−→ ∇f (x) +∇c(x)λ = 0c(x) = 0

I System of nonlinear equations in (x , λ) ∈ Rn × Rm

I Apply Newton’s method: Fast local convergence!I Issues:

I Guarantees only (fast) local convergence.I We would like to find local minima and not just any kind of

stationary point.I Need:

I Globalization scheme (for convergence from any starting point)I Mechanisms that encourage convergence to local minima

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 127: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Equality-Constrained Nonlinear Problems

minx∈Rn

f (x)

s.t. c(x) = 0−→ ∇f (x) +∇c(x)λ = 0

c(x) = 0

I System of nonlinear equations in (x , λ) ∈ Rn × Rm

I Apply Newton’s method: Fast local convergence!I Issues:

I Guarantees only (fast) local convergence.I We would like to find local minima and not just any kind of

stationary point.I Need:

I Globalization scheme (for convergence from any starting point)I Mechanisms that encourage convergence to local minima

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 128: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Equality-Constrained Nonlinear Problems

minx∈Rn

f (x)

s.t. c(x) = 0−→ ∇f (x) +∇c(x)λ = 0

c(x) = 0

I System of nonlinear equations in (x , λ) ∈ Rn × Rm

I Apply Newton’s method: Fast local convergence!I Issues:

I Guarantees only (fast) local convergence.I We would like to find local minima and not just any kind of

stationary point.I Need:

I Globalization scheme (for convergence from any starting point)I Mechanisms that encourage convergence to local minima

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 129: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Equality-Constrained Nonlinear Problems

minx∈Rn

f (x)

s.t. c(x) = 0−→ ∇f (x) +∇c(x)λ = 0

c(x) = 0

I System of nonlinear equations in (x , λ) ∈ Rn × Rm

I Apply Newton’s method: Fast local convergence!

I Issues:I Guarantees only (fast) local convergence.I We would like to find local minima and not just any kind of

stationary point.I Need:

I Globalization scheme (for convergence from any starting point)I Mechanisms that encourage convergence to local minima

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 130: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Equality-Constrained Nonlinear Problems

minx∈Rn

f (x)

s.t. c(x) = 0−→ ∇f (x) +∇c(x)λ = 0

c(x) = 0

I System of nonlinear equations in (x , λ) ∈ Rn × Rm

I Apply Newton’s method: Fast local convergence!I Issues:

I Guarantees only (fast) local convergence.

I We would like to find local minima and not just any kind ofstationary point.

I Need:I Globalization scheme (for convergence from any starting point)I Mechanisms that encourage convergence to local minima

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 131: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Equality-Constrained Nonlinear Problems

minx∈Rn

f (x)

s.t. c(x) = 0−→ ∇f (x) +∇c(x)λ = 0

c(x) = 0

I System of nonlinear equations in (x , λ) ∈ Rn × Rm

I Apply Newton’s method: Fast local convergence!I Issues:

I Guarantees only (fast) local convergence.I We would like to find local minima and not just any kind of

stationary point.

I Need:I Globalization scheme (for convergence from any starting point)I Mechanisms that encourage convergence to local minima

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 132: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Equality-Constrained Nonlinear Problems

minx∈Rn

f (x)

s.t. c(x) = 0−→ ∇f (x) +∇c(x)λ = 0

c(x) = 0

I System of nonlinear equations in (x , λ) ∈ Rn × Rm

I Apply Newton’s method: Fast local convergence!I Issues:

I Guarantees only (fast) local convergence.I We would like to find local minima and not just any kind of

stationary point.I Need:

I Globalization scheme (for convergence from any starting point)I Mechanisms that encourage convergence to local minima

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 133: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Newton’s Method

∇f (x) +∇c(x)λ = 0c(x) = 0

At iterate (xk , λk) compute step pk , pλk from

[Hk ∇ck∇cTk 0

](pkpλk

)= −

(∇fk +∇ckλk

ck

)

I Update iterate (xk+1, λk+1) = (xk , λk) + (pk , pλk )

∇fk := ∇f (xk) ∇ck := ∇c(xk) ck := c(xk)

L(x , λ) := f (x) +m∑j=1

cj(x)λj Hk := ∇2xxL(xk , λk)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 134: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Newton’s Method

∇f (x) +∇c(x)λ = 0c(x) = 0

At iterate (xk , λk) compute step pk , pλk from[Hk ∇ck∇cTk 0

](pkpλk

)= −

(∇fk +∇ckλk

ck

)

I Update iterate (xk+1, λk+1) = (xk , λk) + (pk , pλk )

∇fk := ∇f (xk) ∇ck := ∇c(xk) ck := c(xk)

L(x , λ) := f (x) +m∑j=1

cj(x)λj Hk := ∇2xxL(xk , λk)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 135: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Newton’s Method

∇f (x) +∇c(x)λ = 0c(x) = 0

At iterate (xk , λk) compute step pk , pλk from[Hk ∇ck∇cTk 0

](pkpλk

)= −

(∇fk +∇ckλk

ck

)

I Update iterate (xk+1, λk+1) = (xk , λk) + (pk , pλk )

∇fk := ∇f (xk) ∇ck := ∇c(xk) ck := c(xk)

L(x , λ) := f (x) +m∑j=1

cj(x)λj Hk := ∇2xxL(xk , λk)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 136: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Newton’s Method

∇f (x) +∇c(x)λ = 0c(x) = 0

At iterate (xk , λk) compute step pk , pλk from[Hk ∇ck∇cTk 0

](pkpλk

)= −

(∇fk +∇ckλk

ck

)

I Update iterate (xk+1, λk+1) = (xk , λk) + (pk , pλk )

∇fk := ∇f (xk) ∇ck := ∇c(xk) ck := c(xk)

L(x , λ) := f (x) +m∑j=1

cj(x)λj Hk := ∇2xxL(xk , λk)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 137: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Sequential Quadratic Programming

[Hk ∇ck∇cTk 0

](pkpλk

)= −

(∇fk+∇ckλk

ck

)

These are the optimality conditions of

minp∈Rn

12p

THkp +∇f Tk p + fk

s.t. ∇cTk p + ck = 0

with multipliers

λk+1 = λk + pλkI Newton step can be interpreted as solution of a local QP model!

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 138: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Sequential Quadratic Programming

[Hk ∇ck∇cTk 0

](pk

λk + pλk

)= −

(∇fk

+∇ckλkck

)

These are the optimality conditions of

minp∈Rn

12p

THkp +∇f Tk p + fk

s.t. ∇cTk p + ck = 0

with multipliers

λk+1 = λk + pλkI Newton step can be interpreted as solution of a local QP model!

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 139: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Sequential Quadratic Programming

[Hk ∇ck∇cTk 0

](pkλk+1

)= −

(∇fkck

)

These are the optimality conditions of

minp∈Rn

12p

THkp +∇f Tk p + fk

s.t. ∇cTk p + ck = 0

with multipliers

λk+1 = λk + pλk

I Newton step can be interpreted as solution of a local QP model!

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 140: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Sequential Quadratic Programming

[Hk ∇ck∇cTk 0

](pkλk+1

)= −

(∇fkck

)

These are the optimality conditions of

minp∈Rn

12p

THkp +∇f Tk p + fk

s.t. ∇cTk p + ck = 0

with multipliers λk+1 = λk + pλk

I Newton step can be interpreted as solution of a local QP model!

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 141: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Sequential Quadratic Programming

[Hk ∇ck∇cTk 0

](pkλk+1

)= −

(∇fkck

)

These are the optimality conditions of

minp∈Rn

12p

THkp +∇f Tk p + fk

s.t. ∇cTk p + ck = 0

with multipliers λk+1 = λk + pλkI Newton step can be interpreted as solution of a local QP model!

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 142: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Local QP Model

x∗

xk

c(x) = 0

Original Problem

minx∈Rn

f (x)

s.t. c(x) = 0

Local QP model (QPk)

minp∈Rn

fk +∇f Tk p +12p

THkp

s.t. ck +∇cTk p = 0

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 143: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Local QP Model

x∗

xk

c(x) = 0

p

Original Problem

minx∈Rn

f (x)

s.t. c(x) = 0

Local QP model (QPk)

minp∈Rn

fk +∇f Tk p +12p

THkp

s.t. ck +∇cTk p = 0

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 144: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Exact Penalty FunctionI Need tool to facilitate convergence from any starting point.

I Here, we have two (usually competing) goals:

Optimalitymin f (x)

Feasibilitymin ‖c(x)‖

I Combined in (non-differentiable) exact penalty function:

φρ(x) = f (x) + ρ‖c(x)‖1 (ρ > 0)

LemmaSuppose, x∗ is a local minimizer of (NLP) with multipliers λ∗ and LICQholds. Then x∗ is a local minimizer of φρ if ρ > ‖λ∗‖∞.

I We can use decrease in φρ as a measure of progress towards a localminimizer of (NLP).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 145: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Exact Penalty FunctionI Need tool to facilitate convergence from any starting point.I Here, we have two (usually competing) goals:

Optimalitymin f (x)

Feasibilitymin ‖c(x)‖

I Combined in (non-differentiable) exact penalty function:

φρ(x) = f (x) + ρ‖c(x)‖1 (ρ > 0)

LemmaSuppose, x∗ is a local minimizer of (NLP) with multipliers λ∗ and LICQholds. Then x∗ is a local minimizer of φρ if ρ > ‖λ∗‖∞.

I We can use decrease in φρ as a measure of progress towards a localminimizer of (NLP).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 146: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Exact Penalty FunctionI Need tool to facilitate convergence from any starting point.I Here, we have two (usually competing) goals:

Optimalitymin f (x)

Feasibilitymin ‖c(x)‖

I Combined in (non-differentiable) exact penalty function:

φρ(x) = f (x) + ρ‖c(x)‖1 (ρ > 0)

LemmaSuppose, x∗ is a local minimizer of (NLP) with multipliers λ∗ and LICQholds. Then x∗ is a local minimizer of φρ if ρ > ‖λ∗‖∞.

I We can use decrease in φρ as a measure of progress towards a localminimizer of (NLP).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 147: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Exact Penalty FunctionI Need tool to facilitate convergence from any starting point.I Here, we have two (usually competing) goals:

Optimalitymin f (x)

Feasibilitymin ‖c(x)‖

I Combined in (non-differentiable) exact penalty function:

φρ(x) = f (x) + ρ‖c(x)‖1 (ρ > 0)

LemmaSuppose, x∗ is a local minimizer of (NLP) with multipliers λ∗ and LICQholds. Then x∗ is a local minimizer of φρ if ρ > ‖λ∗‖∞.

I We can use decrease in φρ as a measure of progress towards a localminimizer of (NLP).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 148: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Exact Penalty FunctionI Need tool to facilitate convergence from any starting point.I Here, we have two (usually competing) goals:

Optimalitymin f (x)

Feasibilitymin ‖c(x)‖

I Combined in (non-differentiable) exact penalty function:

φρ(x) = f (x) + ρ‖c(x)‖1 (ρ > 0)

LemmaSuppose, x∗ is a local minimizer of (NLP) with multipliers λ∗ and LICQholds. Then x∗ is a local minimizer of φρ if ρ > ‖λ∗‖∞.

I We can use decrease in φρ as a measure of progress towards a localminimizer of (NLP).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 149: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Exact Penalty FunctionI Need tool to facilitate convergence from any starting point.I Here, we have two (usually competing) goals:

Optimalitymin f (x)

Feasibilitymin ‖c(x)‖

I Combined in (non-differentiable) exact penalty function:

φρ(x) = f (x) + ρ‖c(x)‖1 (ρ > 0)

LemmaSuppose, x∗ is a local minimizer of (NLP) with multipliers λ∗ and LICQholds. Then x∗ is a local minimizer of φρ if ρ > ‖λ∗‖∞.

I We can use decrease in φρ as a measure of progress towards a localminimizer of (NLP).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 150: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Line Search

φρ(x) = f (x) + ρ‖c(x)‖1

I Backtracking line search: Try αk ∈ 1, 12 ,

14 , . . . until

φρ(xk + αkpk) ≤ φρ(xk) + ηαkDφρ(xk ; pk). (η ∈ (0, 1))

Dφρ(xk ; pk): Directional derivative of φρ at xk in direction pk .

LemmaLet pk be an optimal solution of (QPk). Then

Dφρ(xk ; pk) ≤ −pTk Hkpk − (ρ− ‖λk+1‖∞)‖ck‖1.

I So, pk is a descent direction for φρ if Hk 0 and ρ > ‖λk+1‖∞.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 151: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Line Search

φρ(x) = f (x) + ρ‖c(x)‖1

I Backtracking line search: Try αk ∈ 1, 12 ,

14 , . . . until

φρ(xk + αkpk) ≤ φρ(xk) + ηαkDφρ(xk ; pk). (η ∈ (0, 1))

Dφρ(xk ; pk): Directional derivative of φρ at xk in direction pk .

LemmaLet pk be an optimal solution of (QPk). Then

Dφρ(xk ; pk) ≤ −pTk Hkpk − (ρ− ‖λk+1‖∞)‖ck‖1.

I So, pk is a descent direction for φρ if Hk 0 and ρ > ‖λk+1‖∞.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 152: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Line Search

φρ(x) = f (x) + ρ‖c(x)‖1

I Backtracking line search: Try αk ∈ 1, 12 ,

14 , . . . until

φρ(xk + αkpk) ≤ φρ(xk) + ηαkDφρ(xk ; pk). (η ∈ (0, 1))

Dφρ(xk ; pk): Directional derivative of φρ at xk in direction pk .

LemmaLet pk be an optimal solution of (QPk). Then

Dφρ(xk ; pk) ≤ −pTk Hkpk − (ρ− ‖λk+1‖∞)‖ck‖1.

I So, pk is a descent direction for φρ if Hk 0 and ρ > ‖λk+1‖∞.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 153: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Line Search

φρ(x) = f (x) + ρ‖c(x)‖1

I Backtracking line search: Try αk ∈ 1, 12 ,

14 , . . . until

φρ(xk + αkpk) ≤ φρ(xk) + ηαkDφρ(xk ; pk). (η ∈ (0, 1))

Dφρ(xk ; pk): Directional derivative of φρ at xk in direction pk .

LemmaLet pk be an optimal solution of (QPk). Then

Dφρ(xk ; pk) ≤ −pTk Hkpk − (ρ− ‖λk+1‖∞)‖ck‖1.

I So, pk is a descent direction for φρ if Hk 0 and ρ > ‖λk+1‖∞.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 154: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Line Search

φρ(x) = f (x) + ρ‖c(x)‖1

I Backtracking line search: Try αk ∈ 1, 12 ,

14 , . . . until

φρ(xk + αkpk) ≤ φρ(xk) + ηαkDφρ(xk ; pk). (η ∈ (0, 1))

Dφρ(xk ; pk): Directional derivative of φρ at xk in direction pk .

LemmaLet pk be an optimal solution of (QPk). Then

Dφρ(xk ; pk) ≤ −pTk Hkpk − (ρ− ‖λk+1‖∞)‖ck‖1.

I So, pk is a descent direction for φρ if Hk 0 and ρ > ‖λk+1‖∞.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 155: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Line Search

φρ(x) = f (x) + ρ‖c(x)‖1

I Backtracking line search: Try αk ∈ 1, 12 ,

14 , . . . until

φρ(xk + αkpk) ≤ φρ(xk) + ηαkDφρ(xk ; pk). (η ∈ (0, 1))

Dφρ(xk ; pk): Directional derivative of φρ at xk in direction pk .

LemmaLet pk be an optimal solution of (QPk). Then

Dφρ(xk ; pk) ≤ −pTk Hkpk − (ρ− ‖λk+1‖∞)‖ck‖1.

I So, pk is a descent direction for φρ if

Hk 0 and ρ > ‖λk+1‖∞.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 156: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Line Search

φρ(x) = f (x) + ρ‖c(x)‖1

I Backtracking line search: Try αk ∈ 1, 12 ,

14 , . . . until

φρ(xk + αkpk) ≤ φρ(xk) + ηαkDφρ(xk ; pk). (η ∈ (0, 1))

Dφρ(xk ; pk): Directional derivative of φρ at xk in direction pk .

LemmaLet pk be an optimal solution of (QPk). Then

Dφρ(xk ; pk) ≤ −pTk Hkpk − (ρ− ‖λk+1‖∞)‖ck‖1.

I So, pk is a descent direction for φρ if Hk 0 and ρ > ‖λk+1‖∞.Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 157: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Basic SQP Algorithm

1. Choose x1, λ1, ρ0 > 0. Set k ← 1.

2. Solve (QPk) to get pk and λk+1.3. Update penalty parameter: (β > 0)

ρk =

ρk−1 if ρk−1 ≥ ‖λk+1‖∞ + β

‖λk+1‖∞ + 2β otherwise.

4. Perform backtracking line search:Find largest αk ∈ 1, 1

2 ,14 , . . . with

φρk (xk + αkpk) ≤ φρ(xk) + ηαkDφρk (xk ; pk).

5. Update iterate xk+1 = xk + αkpk and λk+1 = λk+1.6. Set k ← k + 1 and to go Step 2.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 158: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Basic SQP Algorithm

1. Choose x1, λ1, ρ0 > 0. Set k ← 1.2. Solve (QPk) to get pk and λk+1.

3. Update penalty parameter: (β > 0)

ρk =

ρk−1 if ρk−1 ≥ ‖λk+1‖∞ + β

‖λk+1‖∞ + 2β otherwise.

4. Perform backtracking line search:Find largest αk ∈ 1, 1

2 ,14 , . . . with

φρk (xk + αkpk) ≤ φρ(xk) + ηαkDφρk (xk ; pk).

5. Update iterate xk+1 = xk + αkpk and λk+1 = λk+1.6. Set k ← k + 1 and to go Step 2.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 159: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Basic SQP Algorithm

1. Choose x1, λ1, ρ0 > 0. Set k ← 1.2. Solve (QPk) to get pk and λk+1.3. Update penalty parameter: (β > 0)

ρk =

ρk−1 if ρk−1 ≥ ‖λk+1‖∞ + β

‖λk+1‖∞ + 2β otherwise.

4. Perform backtracking line search:Find largest αk ∈ 1, 1

2 ,14 , . . . with

φρk (xk + αkpk) ≤ φρ(xk) + ηαkDφρk (xk ; pk).

5. Update iterate xk+1 = xk + αkpk and λk+1 = λk+1.6. Set k ← k + 1 and to go Step 2.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 160: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Basic SQP Algorithm

1. Choose x1, λ1, ρ0 > 0. Set k ← 1.2. Solve (QPk) to get pk and λk+1.3. Update penalty parameter: (β > 0)

ρk =

ρk−1 if ρk−1 ≥ ‖λk+1‖∞ + β

‖λk+1‖∞ + 2β otherwise.

4. Perform backtracking line search:Find largest αk ∈ 1, 1

2 ,14 , . . . with

φρk (xk + αkpk) ≤ φρ(xk) + ηαkDφρk (xk ; pk).

5. Update iterate xk+1 = xk + αkpk and λk+1 = λk+1.6. Set k ← k + 1 and to go Step 2.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 161: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Basic SQP Algorithm

1. Choose x1, λ1, ρ0 > 0. Set k ← 1.2. Solve (QPk) to get pk and λk+1.3. Update penalty parameter: (β > 0)

ρk =

ρk−1 if ρk−1 ≥ ‖λk+1‖∞ + β

‖λk+1‖∞ + 2β otherwise.

4. Perform backtracking line search:Find largest αk ∈ 1, 1

2 ,14 , . . . with

φρk (xk + αkpk) ≤ φρ(xk) + ηαkDφρk (xk ; pk).

5. Update iterate xk+1 = xk + αkpk and λk+1 = λk+1.

6. Set k ← k + 1 and to go Step 2.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 162: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Basic SQP Algorithm

1. Choose x1, λ1, ρ0 > 0. Set k ← 1.2. Solve (QPk) to get pk and λk+1.3. Update penalty parameter: (β > 0)

ρk =

ρk−1 if ρk−1 ≥ ‖λk+1‖∞ + β

‖λk+1‖∞ + 2β otherwise.

4. Perform backtracking line search:Find largest αk ∈ 1, 1

2 ,14 , . . . with

φρk (xk + αkpk) ≤ φρ(xk) + ηαkDφρk (xk ; pk).

5. Update iterate xk+1 = xk + αkpk and λk+1 = λk+1.6. Set k ← k + 1 and to go Step 2.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 163: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Convergence Result for Basic SQP Algorithm

AssumptionsI f and c are twice continuously differentiable.

I The matrices Hk are bounded and uniformly positive definite.I The smallest singular value of ∇ck is uniformly bounded away from

zero.

TheoremUnder these assumptions, we have

limk→∞

∥∥∥∥∥(∇fk +∇ck λk+1

ck

)∥∥∥∥∥ = 0.

In other words, each limit point of xk is a stationary point for (NLP).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 164: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Convergence Result for Basic SQP Algorithm

AssumptionsI f and c are twice continuously differentiable.I The matrices Hk are bounded and uniformly positive definite.

I The smallest singular value of ∇ck is uniformly bounded away fromzero.

TheoremUnder these assumptions, we have

limk→∞

∥∥∥∥∥(∇fk +∇ck λk+1

ck

)∥∥∥∥∥ = 0.

In other words, each limit point of xk is a stationary point for (NLP).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 165: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Convergence Result for Basic SQP Algorithm

AssumptionsI f and c are twice continuously differentiable.I The matrices Hk are bounded and uniformly positive definite.I The smallest singular value of ∇ck is uniformly bounded away from

zero.

TheoremUnder these assumptions, we have

limk→∞

∥∥∥∥∥(∇fk +∇ck λk+1

ck

)∥∥∥∥∥ = 0.

In other words, each limit point of xk is a stationary point for (NLP).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 166: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Convergence Result for Basic SQP Algorithm

AssumptionsI f and c are twice continuously differentiable.I The matrices Hk are bounded and uniformly positive definite.I The smallest singular value of ∇ck is uniformly bounded away from

zero.

TheoremUnder these assumptions, we have

limk→∞

∥∥∥∥∥(∇fk +∇ck λk+1

ck

)∥∥∥∥∥ = 0.

In other words, each limit point of xk is a stationary point for (NLP).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 167: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Convergence Result for Basic SQP Algorithm

AssumptionsI f and c are twice continuously differentiable.I The matrices Hk are bounded and uniformly positive definite.I The smallest singular value of ∇ck is uniformly bounded away from

zero.

TheoremUnder these assumptions, we have

limk→∞

∥∥∥∥∥(∇fk +∇ck λk+1

ck

)∥∥∥∥∥ = 0.

In other words, each limit point of xk is a stationary point for (NLP).

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 168: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Choice of Hessian Hk

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk p + ck = 0(QPk)

I For fast local convergence, want to choose Hk = ∇2xxLk .

I ∇2xxLk is positive definite, if f and c are convex.

I In general, ∇2xxLk might be indefinite.

I (QPk) might be unbounded.I pk might not be a descent direction for φρ.I Would like ZT

k HkZk 0 (Zk null-space matrix for ∇cTk ).I Hk = BFGS approximation of ∇2

xxL(xk , λk).I Potentially slow local convergence, since ∇2

xxL(x∗, λ∗) may beindefinite.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 169: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Choice of Hessian Hk

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk p + ck = 0(QPk)

I For fast local convergence, want to choose Hk = ∇2xxLk .

I ∇2xxLk is positive definite, if f and c are convex.

I In general, ∇2xxLk might be indefinite.

I (QPk) might be unbounded.I pk might not be a descent direction for φρ.I Would like ZT

k HkZk 0 (Zk null-space matrix for ∇cTk ).I Hk = BFGS approximation of ∇2

xxL(xk , λk).I Potentially slow local convergence, since ∇2

xxL(x∗, λ∗) may beindefinite.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 170: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Choice of Hessian Hk

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk p + ck = 0(QPk)

I For fast local convergence, want to choose Hk = ∇2xxLk .

I ∇2xxLk is positive definite, if f and c are convex.

I In general, ∇2xxLk might be indefinite.

I (QPk) might be unbounded.I pk might not be a descent direction for φρ.I Would like ZT

k HkZk 0 (Zk null-space matrix for ∇cTk ).I Hk = BFGS approximation of ∇2

xxL(xk , λk).I Potentially slow local convergence, since ∇2

xxL(x∗, λ∗) may beindefinite.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 171: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Choice of Hessian Hk

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk p + ck = 0(QPk)

I For fast local convergence, want to choose Hk = ∇2xxLk .

I ∇2xxLk is positive definite, if f and c are convex.

I In general, ∇2xxLk might be indefinite.

I (QPk) might be unbounded.

I pk might not be a descent direction for φρ.I Would like ZT

k HkZk 0 (Zk null-space matrix for ∇cTk ).I Hk = BFGS approximation of ∇2

xxL(xk , λk).I Potentially slow local convergence, since ∇2

xxL(x∗, λ∗) may beindefinite.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 172: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Choice of Hessian Hk

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk p + ck = 0(QPk)

I For fast local convergence, want to choose Hk = ∇2xxLk .

I ∇2xxLk is positive definite, if f and c are convex.

I In general, ∇2xxLk might be indefinite.

I (QPk) might be unbounded.I pk might not be a descent direction for φρ.

I Would like ZTk HkZk 0 (Zk null-space matrix for ∇cTk ).

I Hk = BFGS approximation of ∇2xxL(xk , λk).

I Potentially slow local convergence, since ∇2xxL(x∗, λ∗) may be

indefinite.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 173: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Choice of Hessian Hk

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk p + ck = 0(QPk)

I For fast local convergence, want to choose Hk = ∇2xxLk .

I ∇2xxLk is positive definite, if f and c are convex.

I In general, ∇2xxLk might be indefinite.

I (QPk) might be unbounded.I pk might not be a descent direction for φρ.I Would like ZT

k HkZk 0 (Zk null-space matrix for ∇cTk ).

I Hk = BFGS approximation of ∇2xxL(xk , λk).

I Potentially slow local convergence, since ∇2xxL(x∗, λ∗) may be

indefinite.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 174: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Choice of Hessian Hk

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk p + ck = 0(QPk)

I For fast local convergence, want to choose Hk = ∇2xxLk .

I ∇2xxLk is positive definite, if f and c are convex.

I In general, ∇2xxLk might be indefinite.

I (QPk) might be unbounded.I pk might not be a descent direction for φρ.I Would like ZT

k HkZk 0 (Zk null-space matrix for ∇cTk ).I Hk = BFGS approximation of ∇2

xxL(xk , λk).

I Potentially slow local convergence, since ∇2xxL(x∗, λ∗) may be

indefinite.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 175: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Choice of Hessian Hk

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk p + ck = 0(QPk)

I For fast local convergence, want to choose Hk = ∇2xxLk .

I ∇2xxLk is positive definite, if f and c are convex.

I In general, ∇2xxLk might be indefinite.

I (QPk) might be unbounded.I pk might not be a descent direction for φρ.I Would like ZT

k HkZk 0 (Zk null-space matrix for ∇cTk ).I Hk = BFGS approximation of ∇2

xxL(xk , λk).I Potentially slow local convergence, since ∇2

xxL(x∗, λ∗) may beindefinite.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 176: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Regularization

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk p + ck = 0

Recall optimality conditions[Hk ∇ck∇cTk 0

]︸ ︷︷ ︸

=:Kk

(pkλk+1

)= −

(∇fkck

)

I Choose γ ≥ 0 so that Kk has inertia (n,m, 0).

I E.g.: Trial and error, computing inertia via factorizationI Then ZT

k HkZk is positive definite.I No regularization necessary close to second-order sufficient solution.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 177: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Regularization

minp∈Rn

12p

T (Hk + γI)p +∇f Tk p

s.t. ∇cTk p + ck = 0

Recall optimality conditions[Hk + γI ∇ck∇cTk 0

]︸ ︷︷ ︸

=:Kk

(pkλk+1

)= −

(∇fkck

)

I Choose γ ≥ 0

so that Kk has inertia (n,m, 0).

I E.g.: Trial and error, computing inertia via factorizationI Then ZT

k HkZk is positive definite.I No regularization necessary close to second-order sufficient solution.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 178: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Regularization

minp∈Rn

12p

T (Hk + γI)p +∇f Tk p

s.t. ∇cTk p + ck = 0

Recall optimality conditions[Hk + γI ∇ck∇cTk 0

]︸ ︷︷ ︸

=:Kk

(pkλk+1

)= −

(∇fkck

)

I Choose γ ≥ 0 so that Kk has inertia (n,m, 0).

I E.g.: Trial and error, computing inertia via factorization

I Then ZTk HkZk is positive definite.

I No regularization necessary close to second-order sufficient solution.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 179: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Regularization

minp∈Rn

12p

T (Hk + γI)p +∇f Tk p

s.t. ∇cTk p + ck = 0

Recall optimality conditions[Hk + γI ∇ck∇cTk 0

]︸ ︷︷ ︸

=:Kk

(pkλk+1

)= −

(∇fkck

)

I Choose γ ≥ 0 so that Kk has inertia (n,m, 0).I E.g.: Trial and error, computing inertia via factorization

I Then ZTk HkZk is positive definite.

I No regularization necessary close to second-order sufficient solution.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 180: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Regularization

minp∈Rn

12p

T (Hk + γI)p +∇f Tk p

s.t. ∇cTk p + ck = 0

Recall optimality conditions[Hk + γI ∇ck∇cTk 0

]︸ ︷︷ ︸

=:Kk

(pkλk+1

)= −

(∇fkck

)

I Choose γ ≥ 0 so that Kk has inertia (n,m, 0).I E.g.: Trial and error, computing inertia via factorization

I Then ZTk HkZk is positive definite.

I No regularization necessary close to second-order sufficient solution.Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 181: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Step Decomposition Revisited

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk p + ck = 0(QPk)

Decomposition pk = YkpY ,k + ZkpZ ,k

I Range space steppY ,k = −[∇cTk Yk ]−1∇fk

I Reduced space QP

minpZ

12p

TZ [ZT

k HkZk ]pZ + (∇fk + HkYkpY ,k)TZkpZ

I Make sure ZTk HkZk is positive definite (e.g., BFGS)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 182: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Step Decomposition Revisited

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk p + ck = 0(QPk)

Decomposition pk = YkpY ,k + ZkpZ ,kI Range space step

pY ,k = −[∇cTk Yk ]−1∇fk

I Reduced space QP

minpZ

12p

TZ [ZT

k HkZk ]pZ + (∇fk + HkYkpY ,k)TZkpZ

I Make sure ZTk HkZk is positive definite (e.g., BFGS)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 183: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Step Decomposition Revisited

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk p + ck = 0(QPk)

Decomposition pk = YkpY ,k + ZkpZ ,kI Range space step

pY ,k = −[∇cTk Yk ]−1∇fk

I Reduced space QP

minpZ

12p

TZ [ZT

k HkZk ]pZ + (∇fk + HkYkpY ,k)TZkpZ

I Make sure ZTk HkZk is positive definite (e.g., BFGS)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 184: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Step Decomposition Revisited

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk p + ck = 0(QPk)

Decomposition pk = YkpY ,k + ZkpZ ,kI Range space step

pY ,k = −[∇cTk Yk ]−1∇fk

I Reduced space QP

minpZ

12p

TZ [ZT

k HkZk ]pZ + (∇fk + HkYkpY ,k)TZkpZ

I Make sure ZTk HkZk is positive definite

(e.g., BFGS)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 185: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Step Decomposition Revisited

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk p + ck = 0(QPk)

Decomposition pk = YkpY ,k + ZkpZ ,kI Range space step

pY ,k = −[∇cTk Yk ]−1∇fk

I Reduced space QP

minpZ

12p

TZ [ZT

k HkZk ]pZ + (∇fk + HkYkpY ,k)TZkpZ

I Make sure ZTk HkZk is positive definite (e.g., BFGS)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 186: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Trust-Region SQP Method

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk p + ck = 0, ‖p‖ ≤ ∆k

(QPk)

I Trust-region radius ∆k , updated throughout iterations

I No positive-definiteness requirements for HkI Step pk is accepted if predk

aredk ≥ η with (η ∈ (0, 1))

predk = mk(0)−mk(pk), aredk = φρ(xk)− φρ(xk + pk)

I Piece-wise quadratic model of φρ(x) = f (x) + ρ‖c(x)‖1:

mk(p) = fk +∇f Tk p +12p

THkp + ρ‖ck +∇cTk p‖1

I Otherwise, decrease ∆k

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 187: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Trust-Region SQP Method

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk p + ck = 0, ‖p‖ ≤ ∆k

(QPk)

I Trust-region radius ∆k , updated throughout iterationsI No positive-definiteness requirements for Hk

I Step pk is accepted if predkaredk ≥ η with (η ∈ (0, 1))

predk = mk(0)−mk(pk), aredk = φρ(xk)− φρ(xk + pk)

I Piece-wise quadratic model of φρ(x) = f (x) + ρ‖c(x)‖1:

mk(p) = fk +∇f Tk p +12p

THkp + ρ‖ck +∇cTk p‖1

I Otherwise, decrease ∆k

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 188: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Trust-Region SQP Method

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk p + ck = 0, ‖p‖ ≤ ∆k

(QPk)

I Trust-region radius ∆k , updated throughout iterationsI No positive-definiteness requirements for HkI Step pk is accepted if predk

aredk ≥ η with (η ∈ (0, 1))

predk = mk(0)−mk(pk), aredk = φρ(xk)− φρ(xk + pk)

I Piece-wise quadratic model of φρ(x) = f (x) + ρ‖c(x)‖1:

mk(p) = fk +∇f Tk p +12p

THkp + ρ‖ck +∇cTk p‖1

I Otherwise, decrease ∆k

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 189: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Trust-Region SQP Method

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk p + ck = 0, ‖p‖ ≤ ∆k

(QPk)

I Trust-region radius ∆k , updated throughout iterationsI No positive-definiteness requirements for HkI Step pk is accepted if predk

aredk ≥ η with (η ∈ (0, 1))

predk = mk(0)−mk(pk), aredk = φρ(xk)− φρ(xk + pk)

I Piece-wise quadratic model of φρ(x) = f (x) + ρ‖c(x)‖1:

mk(p) = fk +∇f Tk p +12p

THkp + ρ‖ck +∇cTk p‖1

I Otherwise, decrease ∆k

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 190: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Trust-Region SQP Method

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk p + ck = 0, ‖p‖ ≤ ∆k

(QPk)

I Trust-region radius ∆k , updated throughout iterationsI No positive-definiteness requirements for HkI Step pk is accepted if predk

aredk ≥ η with (η ∈ (0, 1))

predk = mk(0)−mk(pk), aredk = φρ(xk)− φρ(xk + pk)

I Piece-wise quadratic model of φρ(x) = f (x) + ρ‖c(x)‖1:

mk(p) = fk +∇f Tk p +12p

THkp + ρ‖ck +∇cTk p‖1

I Otherwise, decrease ∆kAndreas Wächter Constrained Nonlinear Optimization Algorithms

Page 191: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Inconsistent QPs

x∗

xk

c(x) = 0

p

I If xk is not feasible and ∆k small, (QPk) might not be feasible

I One remedy: Penalize constraint violation

minp∈Rn

;t,s∈Rm

12p

THkp +∇f Tk p

+ ρm∑j=1

(sj + tj)

s.t. ∇cTk p + ck = 0‖p‖ ≤ ∆k

, s, t ≥ 0

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 192: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Inconsistent QPs

x∗

xk

c(x) = 0

p

I If xk is not feasible and ∆k small, (QPk) might not be feasibleI One remedy: Penalize constraint violation

minp∈Rn

;t,s∈Rm

12p

THkp +∇f Tk p

+ ρm∑j=1

(sj + tj)

s.t. ∇cTk p + ck = 0‖p‖ ≤ ∆k

, s, t ≥ 0

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 193: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Inconsistent QPs

x∗

xk

c(x) = 0

p

I If xk is not feasible and ∆k small, (QPk) might not be feasibleI One remedy: Penalize constraint violation

minp∈Rn;t,s∈Rm

12p

THkp +∇f Tk p + ρm∑j=1

(sj + tj)

s.t. ∇cTk p + ck = s − t‖p‖ ≤ ∆k , s, t ≥ 0

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 194: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Fletcher’s S`1QP

minp∈Rn;t,s∈Rm

12p

THkp +∇f Tk p + ρm∑j=1

(sj + tj)

s.t. ∇cTk p + ck = s − t‖p‖ ≤ ∆k , s, t ≥ 0

is equivalent to

minp∈Rn

mk(p) = fk +∇f Tk p +12p

THkp + ρ‖ck +∇cTk p‖1

I Natural algorithm for minimizing φρ(x).I Difficulty: Selecting sufficiently large value of ρ.

I This motivated the invention of filter methods.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 195: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Fletcher’s S`1QP

minp∈Rn;t,s∈Rm

12p

THkp +∇f Tk p + ρm∑j=1

(sj + tj)

s.t. ∇cTk p + ck = s − t‖p‖ ≤ ∆k , s, t ≥ 0

is equivalent to

minp∈Rn

mk(p) = fk +∇f Tk p +12p

THkp + ρ‖ck +∇cTk p‖1

I Natural algorithm for minimizing φρ(x).I Difficulty: Selecting sufficiently large value of ρ.

I This motivated the invention of filter methods.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 196: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Fletcher’s S`1QP

minp∈Rn;t,s∈Rm

12p

THkp +∇f Tk p + ρm∑j=1

(sj + tj)

s.t. ∇cTk p + ck = s − t‖p‖ ≤ ∆k , s, t ≥ 0

is equivalent to

minp∈Rn

mk(p) = fk +∇f Tk p +12p

THkp + ρ‖ck +∇cTk p‖1

I Natural algorithm for minimizing φρ(x).

I Difficulty: Selecting sufficiently large value of ρ.I This motivated the invention of filter methods.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 197: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Fletcher’s S`1QP

minp∈Rn;t,s∈Rm

12p

THkp +∇f Tk p + ρm∑j=1

(sj + tj)

s.t. ∇cTk p + ck = s − t‖p‖ ≤ ∆k , s, t ≥ 0

is equivalent to

minp∈Rn

mk(p) = fk +∇f Tk p +12p

THkp + ρ‖ck +∇cTk p‖1

I Natural algorithm for minimizing φρ(x).I Difficulty: Selecting sufficiently large value of ρ.

I This motivated the invention of filter methods.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 198: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Fletcher’s S`1QP

minp∈Rn;t,s∈Rm

12p

THkp +∇f Tk p + ρm∑j=1

(sj + tj)

s.t. ∇cTk p + ck = s − t‖p‖ ≤ ∆k , s, t ≥ 0

is equivalent to

minp∈Rn

mk(p) = fk +∇f Tk p +12p

THkp + ρ‖ck +∇cTk p‖1

I Natural algorithm for minimizing φρ(x).I Difficulty: Selecting sufficiently large value of ρ.

I This motivated the invention of filter methods.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 199: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Byrd-Omojokun Trust-Region Algorithm

x∗

xk

nk

I Decompose step pk = nk + tk

Normal component nktowards feasibility

minn‖∇cTk n + ck‖22

s.t. ‖n‖2 ≤ 0.8∆k

Tangential component tktowards optimality

mint

12(nk +t)THk(nk +t) +∇f Tk (nk +t)

s.t. ∇cTk t = 0

‖t‖2 ≤√

∆2k − ‖nk‖22

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 200: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Byrd-Omojokun Trust-Region Algorithm

x∗

xk

nk

I Decompose step pk = nk + tkNormal component nktowards feasibility

minn‖∇cTk n + ck‖22

s.t. ‖n‖2 ≤ 0.8∆k

Tangential component tktowards optimality

mint

12(nk +t)THk(nk +t) +∇f Tk (nk +t)

s.t. ∇cTk t = 0

‖t‖2 ≤√

∆2k − ‖nk‖22

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 201: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Byrd-Omojokun Trust-Region Algorithm

x∗

xkpk

tknk

I Decompose step pk = nk + tkNormal component nktowards feasibility

minn‖∇cTk n + ck‖22

s.t. ‖n‖2 ≤ 0.8∆k

Tangential component tktowards optimality

mint

12(nk +t)THk(nk +t) +∇f Tk (nk +t)

s.t. ∇cTk t = 0

‖t‖2 ≤√

∆2k − ‖nk‖22

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 202: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Byrd-Omojokun Trust-Region Algorithm

x∗

xkpk

tknk

I Decompose step pk = nk + tkNormal component nktowards feasibility

minn‖∇cTk n + ck‖22

s.t. ‖n‖2 ≤ 0.8∆k

Tangential component tktowards optimality

mint

12(nk +t)THk(nk +t) +∇f Tk (nk +t)

s.t. ∇cTk t = 0

‖t‖2 ≤√

∆2k − ‖nk‖22

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 203: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Byrd-Omojokun Trust-Region Algorithm

x∗

xkpk

tknk

I Decompose step pk = nk + tkNormal component nktowards feasibility

minn‖∇cTk n + ck‖22

s.t. ‖n‖2 ≤ 0.8∆k

Tangential component tktowards optimality

mint

12(nk +t)THk(nk +t) +∇f Tk (nk +t)

s.t. ∇cTk t = 0

‖t‖2 ≤√

∆2k − ‖nk‖22

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 204: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Byrd-Omojokun Trust-Region Algorithm

Normal component nktowards feasibility

minn‖∇cTk n + ck‖22

s.t. ‖n‖2 ≤ 0.8∆k

Tangential component tktowards optimality

mint

12(nk +t)THk(nk +t) +∇f Tk (nk +t)

s.t. ∇cTk t = 0

‖t‖2 ≤√

∆2k − ‖nk‖22

I Subproblems can be solved inexactly

I Normal problem: Dogleg method.I Tangential problem: Conjugate-gradients method in null space.

I `2-norm penalty function φρ(x) = f (x) + ρ‖c(x)‖2.I Strong convergence result:

I If (NLP) is infeasible, limit points of xk are stationary points forinfeasibility minimization problem minx ‖c(x)‖2

2.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 205: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Byrd-Omojokun Trust-Region Algorithm

Normal component nktowards feasibility

minn‖∇cTk n + ck‖22

s.t. ‖n‖2 ≤ 0.8∆k

Tangential component tktowards optimality

mint

12(nk +t)THk(nk +t) +∇f Tk (nk +t)

s.t. ∇cTk t = 0

‖t‖2 ≤√

∆2k − ‖nk‖22

I Subproblems can be solved inexactlyI Normal problem: Dogleg method.

I Tangential problem: Conjugate-gradients method in null space.I `2-norm penalty function φρ(x) = f (x) + ρ‖c(x)‖2.I Strong convergence result:

I If (NLP) is infeasible, limit points of xk are stationary points forinfeasibility minimization problem minx ‖c(x)‖2

2.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 206: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Byrd-Omojokun Trust-Region Algorithm

Normal component nktowards feasibility

minn‖∇cTk n + ck‖22

s.t. ‖n‖2 ≤ 0.8∆k

Tangential component tktowards optimality

mint

12(nk +t)THk(nk +t) +∇f Tk (nk +t)

s.t. ∇cTk t = 0

‖t‖2 ≤√

∆2k − ‖nk‖22

I Subproblems can be solved inexactlyI Normal problem: Dogleg method.I Tangential problem: Conjugate-gradients method in null space.

I `2-norm penalty function φρ(x) = f (x) + ρ‖c(x)‖2.I Strong convergence result:

I If (NLP) is infeasible, limit points of xk are stationary points forinfeasibility minimization problem minx ‖c(x)‖2

2.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 207: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Byrd-Omojokun Trust-Region Algorithm

Normal component nktowards feasibility

minn‖∇cTk n + ck‖22

s.t. ‖n‖2 ≤ 0.8∆k

Tangential component tktowards optimality

mint

12(nk +t)THk(nk +t) +∇f Tk (nk +t)

s.t. ∇cTk t = 0

‖t‖2 ≤√

∆2k − ‖nk‖22

I Subproblems can be solved inexactlyI Normal problem: Dogleg method.I Tangential problem: Conjugate-gradients method in null space.

I `2-norm penalty function φρ(x) = f (x) + ρ‖c(x)‖2.

I Strong convergence result:I If (NLP) is infeasible, limit points of xk are stationary points for

infeasibility minimization problem minx ‖c(x)‖22.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 208: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Byrd-Omojokun Trust-Region Algorithm

Normal component nktowards feasibility

minn‖∇cTk n + ck‖22

s.t. ‖n‖2 ≤ 0.8∆k

Tangential component tktowards optimality

mint

12(nk +t)THk(nk +t) +∇f Tk (nk +t)

s.t. ∇cTk t = 0

‖t‖2 ≤√

∆2k − ‖nk‖22

I Subproblems can be solved inexactlyI Normal problem: Dogleg method.I Tangential problem: Conjugate-gradients method in null space.

I `2-norm penalty function φρ(x) = f (x) + ρ‖c(x)‖2.I Strong convergence result:

I If (NLP) is infeasible, limit points of xk are stationary points forinfeasibility minimization problem minx ‖c(x)‖2

2.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 209: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Byrd-Omojokun Trust-Region Algorithm

Normal component nktowards feasibility

minn‖∇cTk n + ck‖22

s.t. ‖n‖2 ≤ 0.8∆k

Tangential component tktowards optimality

mint

12(nk +t)THk(nk +t) +∇f Tk (nk +t)

s.t. ∇cTk t = 0

‖t‖2 ≤√

∆2k − ‖nk‖22

I Subproblems can be solved inexactlyI Normal problem: Dogleg method.I Tangential problem: Conjugate-gradients method in null space.

I `2-norm penalty function φρ(x) = f (x) + ρ‖c(x)‖2.I Strong convergence result:

I If (NLP) is infeasible, limit points of xk are stationary points forinfeasibility minimization problem minx ‖c(x)‖2

2.Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 210: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Maratos Effect

I Even arbitrarily close to solution, full step α = 1 might be rejectedbecause the non-smooth merit function φρ increases.

I Degrades fast local convergence.

I Remedies: Second-order correction steps or “watchdog” method.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 211: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Table of Contents

Applications

Equality-Constrained Quadratic Programming

Active-Set Quadratic Programming Solvers

SQP for Equality-Constrained NLPs

SQP for Inequality-Constrained NLPs

Interior Point Methods

Software

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 212: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

SQP For Inequality-Constrained Nonlinear Problems

minx∈Rn

f (x)

s.t. cE (x) = 0

cI(x) ≤ 0

Compute pk from local QP model

minp∈Rn

12p

THkp +∇f (xk)Tp

s.t. ∇cE (xk)Tp + cE (xk) = 0

∇cI(xk)Tp + cI(xk) ≤ 0

(QPk)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 213: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

SQP For Inequality-Constrained Nonlinear Problems

minx∈Rn

f (x)

s.t. cE (x) = 0cI(x) ≤ 0

Compute pk from local QP model

minp∈Rn

12p

THkp +∇f (xk)Tp

s.t. ∇cE (xk)Tp + cE (xk) = 0

∇cI(xk)Tp + cI(xk) ≤ 0

(QPk)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 214: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

SQP For Inequality-Constrained Nonlinear Problems

minx∈Rn

f (x)

s.t. cE (x) = 0cI(x) ≤ 0

Compute pk from local QP model

minp∈Rn

12p

THkp +∇f (xk)Tp

s.t. ∇cE (xk)Tp + cE (xk) = 0∇cI(xk)Tp + cI(xk) ≤ 0

(QPk)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 215: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Local Behavior

minx∈Rn

f (x)

s.t. ci(x) = 0 i ∈ Eci(x) ≤ 0 i ∈ I

ci(x) ≤ 0 i ∈ ANLP∗

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk,ip + ck,i = 0 i ∈ E∇cTk,ip + ck,i = 0 i ∈ AQPk

((((((((

(∇cTk,ip + ck,i ≤ 0 i ∈ AQPk

ANLP∗ = i ∈ I : ci(x∗) = 0 ANLP

∗ = i ∈ I : ci(x∗) < 0

LemmaSuppose x∗ is a local minimizer satisfying the sufficient second-orderoptimality conditions, at which LICQ and strict optimality hold. ThenANLP∗ = AQPk∗ for all xk sufficiently close to x∗.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 216: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Local Behavior

minx∈Rn

f (x)

s.t. ci(x) = 0 i ∈ Eci(x) ≤ 0 i ∈ ANLP

ci(x) ≤ 0 i ∈ ANLP∗

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk,ip + ck,i = 0 i ∈ E∇cTk,ip + ck,i = 0 i ∈ AQPk

((((((((

(∇cTk,ip + ck,i ≤ 0 i ∈ AQPk

ANLP∗ = i ∈ I : ci(x∗) = 0 ANLP

∗ = i ∈ I : ci(x∗) < 0

LemmaSuppose x∗ is a local minimizer satisfying the sufficient second-orderoptimality conditions, at which LICQ and strict optimality hold. ThenANLP∗ = AQPk∗ for all xk sufficiently close to x∗.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 217: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Local Behavior

minx∈Rn

f (x)

s.t. ci(x) = 0 i ∈ Eci(x) = 0 i ∈ ANLP

ci(x) ≤ 0 i ∈ ANLP

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk,ip + ck,i = 0 i ∈ E∇cTk,ip + ck,i = 0 i ∈ AQPk

((((((((

(∇cTk,ip + ck,i ≤ 0 i ∈ AQPk

ANLP∗ = i ∈ I : ci(x∗) = 0 ANLP

∗ = i ∈ I : ci(x∗) < 0

LemmaSuppose x∗ is a local minimizer satisfying the sufficient second-orderoptimality conditions, at which LICQ and strict optimality hold. ThenANLP∗ = AQPk∗ for all xk sufficiently close to x∗.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 218: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Local Behavior

minx∈Rn

f (x)

s.t. ci(x) = 0 i ∈ Eci(x) = 0 i ∈ ANLP

ci(x) ≤ 0 i ∈ ANLP

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk,ip + ck,i = 0 i ∈ E∇cTk,ip + ck,i = 0 i ∈ AQPk

((((((((

(∇cTk,ip + ck,i ≤ 0 i ∈ AQPk

ANLP∗ = i ∈ I : ci(x∗) = 0 ANLP

∗ = i ∈ I : ci(x∗) < 0

LemmaSuppose x∗ is a local minimizer satisfying the sufficient second-orderoptimality conditions, at which LICQ and strict optimality hold. ThenANLP∗ = AQPk∗ for all xk sufficiently close to x∗.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 219: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Local Behavior

minx∈Rn

f (x)

s.t. ci(x) = 0 i ∈ Eci(x) = 0 i ∈ ANLP

ci(x) ≤ 0 i ∈ ANLP

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk,ip + ck,i = 0 i ∈ E∇cTk,ip + ck,i = 0 i ∈ AQPk

((((((((

(∇cTk,ip + ck,i ≤ 0 i ∈ AQPk

ANLP∗ = i ∈ I : ci(x∗) = 0 ANLP

∗ = i ∈ I : ci(x∗) < 0

LemmaSuppose x∗ is a local minimizer satisfying the sufficient second-orderoptimality conditions, at which LICQ and strict optimality hold. ThenANLP∗ = AQPk∗ for all xk sufficiently close to x∗.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 220: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Back to Newton’s Method

minx∈Rn

f (x)

s.t. ci(x) = 0 i ∈ Eci(x) = 0 i ∈ ANLP

ci(x) ≤ 0 i ∈ ANLP

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk,ip + ck,i = 0 i ∈ E∇cTk,ip + ck,i = 0 i ∈ ANLP

(((((((

((∇cTk,ip + ck,i ≤ 0 i ∈ ANLP

I When xk is close to x∗, (QPk) produces the same steps as SQP forequality-constrained NLP.

I We are back to Newton’s method. . .I Fast local convergence!

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 221: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Back to Newton’s Method

minx∈Rn

f (x)

s.t. ci(x) = 0 i ∈ Eci(x) = 0 i ∈ ANLP

ci(x) ≤ 0 i ∈ ANLP

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk,ip + ck,i = 0 i ∈ E∇cTk,ip + ck,i = 0 i ∈ ANLP

(((((((

((∇cTk,ip + ck,i ≤ 0 i ∈ ANLP

I When xk is close to x∗, (QPk) produces the same steps as SQP forequality-constrained NLP.

I We are back to Newton’s method. . .

I Fast local convergence!

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 222: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Back to Newton’s Method

minx∈Rn

f (x)

s.t. ci(x) = 0 i ∈ Eci(x) = 0 i ∈ ANLP

ci(x) ≤ 0 i ∈ ANLP

minp∈Rn

12p

THkp +∇f Tk p

s.t. ∇cTk,ip + ck,i = 0 i ∈ E∇cTk,ip + ck,i = 0 i ∈ ANLP

(((((((

((∇cTk,ip + ck,i ≤ 0 i ∈ ANLP

I When xk is close to x∗, (QPk) produces the same steps as SQP forequality-constrained NLP.

I We are back to Newton’s method. . .I Fast local convergence!

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 223: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Global Convergence

minx∈Rn

f (x)

s.t. cE (x) = 0cI(x) ≤ 0

Methods for equality constraints can be generalized.I For example, penalty function

φρ(x) = f (x) + ρ‖cE (x)‖1 + ρ‖maxcI(x), 0‖1.

minp∈Rn;t,s∈RmE ;r∈RmI

12p

THkp +∇f Tk p + ρmE∑j=1

(sj + tj) + ρmI∑j=1

rj

s.t. ∇cTE ,kp + cE ,k = s − t∇cTE ,kp + cE ,k ≤ r‖p‖ ≤ ∆k , s, t, r ≥ 0

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 224: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Global Convergence

minx∈Rn

f (x)

s.t. cE (x) = 0cI(x) ≤ 0

Methods for equality constraints can be generalized.I For example, penalty function

φρ(x) = f (x) + ρ‖cE (x)‖1 + ρ‖maxcI(x), 0‖1.

minp∈Rn;t,s∈RmE ;r∈RmI

12p

THkp +∇f Tk p + ρmE∑j=1

(sj + tj) + ρmI∑j=1

rj

s.t. ∇cTE ,kp + cE ,k = s − t∇cTE ,kp + cE ,k ≤ r‖p‖ ≤ ∆k , s, t, r ≥ 0

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 225: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Table of Contents

Applications

Equality-Constrained Quadratic Programming

Active-Set Quadratic Programming Solvers

SQP for Equality-Constrained NLPs

SQP for Inequality-Constrained NLPs

Interior Point Methods

Software

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 226: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Barrier Problem

minx∈Rn

f (x)

s.t. c(x) = 0x ≥ 0

−→minx∈Rn

f (x)− µn∑

i=1log(xi)

s.t. c(x) = 0

µ = 10

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 227: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Barrier Problem

minx∈Rn

f (x)

s.t. c(x) = 0x ≥ 0

−→minx∈Rn

f (x)− µn∑

i=1log(xi)

s.t. c(x) = 0

µ = 10

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 228: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Barrier Problem

minx∈R

f (x)

s.t. x ≥ 0

x ≤ 10

−→ minx∈R

f (x)− µ log(x)− µ log(10− x)

µ = 10

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 229: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Barrier Problem

minx∈R

f (x)

s.t. x ≥ 0

x ≤ 10

−→ minx∈R

f (x)− µ log(x)− µ log(10− x)

µ = 1

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 230: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Barrier Problem

minx∈R

f (x)

s.t. x ≥ 0

x ≤ 10

−→ minx∈R

f (x)− µ log(x)− µ log(10− x)

µ = 0.1

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 231: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Barrier Problem

minx∈R

f (x)

s.t. x ≥ 0

x ≤ 10

−→ minx∈R

f (x)− µ log(x)− µ log(10− x)

µ = 0.01

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 232: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Barrier Problem

minx∈R

f (x)

s.t. x ≥ 0

x ≤ 10

−→ minx∈R

f (x)− µ log(x)− µ log(10− x)

µ = 0.001

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 233: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Barrier Method

minx∈Rn

f (x)

s.t. c(x) = 0x ≥ 0

−→minx∈Rn

f (x)− µn∑

i=1log(xi)

s.t. c(x) = 0(BPµ)

Basic Algorithm:1. Choose x0 ∈ Rn, µ0 > 0, ε0 > 0. Set k ← 0.

2. Starting from x0, solve (BPµk ) to tolerance εk and obtain xk+1.

→ Use SQP techniques

3. Decrease µk+1 < µk and εk+1 < εk ; set k ← k + 1; go to 2.

(Ensure µk → 0 and εk → 0.)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 234: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Barrier Method

minx∈Rn

f (x)

s.t. c(x) = 0x ≥ 0

−→minx∈Rn

f (x)− µn∑

i=1log(xi)

s.t. c(x) = 0(BPµ)

Basic Algorithm:1. Choose x0 ∈ Rn, µ0 > 0, ε0 > 0. Set k ← 0.2. Starting from x0, solve (BPµk ) to tolerance εk and obtain xk+1.

→ Use SQP techniques3. Decrease µk+1 < µk and εk+1 < εk ; set k ← k + 1; go to 2.

(Ensure µk → 0 and εk → 0.)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 235: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Barrier Method

minx∈Rn

f (x)

s.t. c(x) = 0x ≥ 0

−→minx∈Rn

f (x)− µn∑

i=1log(xi)

s.t. c(x) = 0(BPµ)

Basic Algorithm:1. Choose x0 ∈ Rn, µ0 > 0, ε0 > 0. Set k ← 0.2. Starting from x0, solve (BPµk ) to tolerance εk and obtain xk+1.

→ Use SQP techniques

3. Decrease µk+1 < µk and εk+1 < εk ; set k ← k + 1; go to 2.

(Ensure µk → 0 and εk → 0.)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 236: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Barrier Method

minx∈Rn

f (x)

s.t. c(x) = 0x ≥ 0

−→minx∈Rn

f (x)− µn∑

i=1log(xi)

s.t. c(x) = 0(BPµ)

Basic Algorithm:1. Choose x0 ∈ Rn, µ0 > 0, ε0 > 0. Set k ← 0.2. Starting from x0, solve (BPµk ) to tolerance εk and obtain xk+1.

→ Use SQP techniques3. Decrease µk+1 < µk and εk+1 < εk ; set k ← k + 1; go to 2.

(Ensure µk → 0 and εk → 0.)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 237: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

SQP Techniques for Solving the Barrier Problem

minx∈Rn

ϕµ(x) = f (x)− µn∑

i=1log(xi)

s.t. c(x) = 0

Can re-use SQP techniques:I Step computation

I KKT system with regularizationI Decomposition

I Step acceptanceI Line searchI Trust region

I Measuring progressI Exact penalty functionI Filter method

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 238: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

SQP Techniques for Solving the Barrier Problem

minx∈Rn

ϕµ(x) = f (x)− µn∑

i=1log(xi)

s.t. c(x) = 0

Can re-use SQP techniques:I Step computation

I KKT system with regularizationI Decomposition

I Step acceptanceI Line searchI Trust region

I Measuring progressI Exact penalty functionI Filter method

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 239: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

SQP Techniques for Solving the Barrier Problem

minx∈Rn

ϕµ(x) = f (x)− µn∑

i=1log(xi)

s.t. c(x) = 0

Can re-use SQP techniques:I Step computation

I KKT system with regularizationI Decomposition

I Step acceptanceI Line searchI Trust region

I Measuring progressI Exact penalty functionI Filter method

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 240: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Barrier Term Considerations

minx∈Rn

ϕµ(x) = f (x)− µn∑

i=1log(xi)

s.t. c(x) = 0

I Variables must stay positive

I Fraction-to-the-boundary rule (τ ∈ (0, 1), e.g., τ = 0.99)

αmaxk = argmax α ∈ (0, 1] : xk + αpk ≥ (1− τ)xk

I Ill-conditioning in linear system

[∇2

xxLk + µX−2k ∇ck

∇cTk 0

](pkλk+1

)= −

(∇fk − µX−1

k eck

)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 241: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Barrier Term Considerations

minx∈Rn

ϕµ(x) = f (x)− µn∑

i=1log(xi)

s.t. c(x) = 0

I Variables must stay positiveI Fraction-to-the-boundary rule (τ ∈ (0, 1), e.g., τ = 0.99)

αmaxk = argmax α ∈ (0, 1] : xk + αpk ≥ (1− τ)xk

I Ill-conditioning in linear system

[∇2

xxLk + µX−2k ∇ck

∇cTk 0

](pkλk+1

)= −

(∇fk − µX−1

k eck

)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 242: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Barrier Term Considerations

minx∈Rn

ϕµ(x) = f (x)− µn∑

i=1log(xi)

s.t. c(x) = 0

I Variables must stay positiveI Fraction-to-the-boundary rule (τ ∈ (0, 1), e.g., τ = 0.99)

αmaxk = argmax α ∈ (0, 1] : xk + αpk ≥ (1− τ)xk

I Ill-conditioning in linear system

[∇2

xxLk + µX−2k ∇ck

∇cTk 0

](pkλk+1

)= −

(∇fk − µX−1

k eck

)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 243: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Barrier Term Considerations

minx∈Rn

ϕµ(x) = f (x)− µn∑

i=1log(xi)

s.t. c(x) = 0

I Variables must stay positiveI Fraction-to-the-boundary rule (τ ∈ (0, 1), e.g., τ = 0.99)

αmaxk = argmax α ∈ (0, 1] : xk + αpk ≥ (1− τ)xk

I Ill-conditioning in linear system[∇2

xxLk + µX−2k ∇ck

∇cTk 0

](pkλk+1

)= −

(∇fk − µX−1

k eck

)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 244: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Example

minx∈R2

x1 + (x2 − 1)2

s.t. x ≥ 0

0 0.5 1 1.5 2 2.5

0

0.5

1

1.5

2

2.5

k | mu_k | f_k | ( x_k(1), x_k(2)) | ( p_k(1), p_k(2)) | err_barr | alpha--------------------------------------------------------------------------------------------0 | 1.00e+00 | 2.50e+00 | (2.00e+00,2.00e+00) | ( 0.00e+00, 0.00e+00) | 7.07e-01 | 0.00e+001 | 1.00e+00 | 2.02e-01 | (2.00e-02,1.60e+00) | (-2.00e+00,-4.00e-01) | 2.04e+00 | 9.90e-012 | 1.00e+00 | 2.31e-01 | (3.96e-02,1.62e+00) | ( 1.96e-02, 1.40e-02) | 2.41e-02 | 1.00e+003 | 1.00e-01 | 6.69e-02 | (6.35e-02,1.08e+00) | ( 2.39e-02,-5.36e-01) | 5.36e-01 | 1.00e+004 | 1.00e-01 | 9.09e-02 | (8.67e-02,1.09e+00) | ( 2.32e-02, 9.33e-03) | 2.50e-02 | 1.00e+005 | 1.00e-02 | 4.15e-03 | (8.67e-04,1.08e+00) | (-6.65e-01,-8.18e-02) | 6.70e-01 | 1.29e-016 | 1.00e-02 | 1.71e-03 | (1.66e-03,1.01e+00) | ( 7.92e-04,-7.12e-02) | 7.12e-02 | 1.00e+00...

12 | 3.16e-05 | 2.70e-05 | (2.70e-05,1.00e+00) | ( 7.45e-06, 2.95e-11) | 7.45e-06 | 1.00e+0013 | 1.78e-07 | 4.98e-10 | (4.81e-12,1.00e+00) | (-4.09e-03,-3.14e-05) | 4.09e-03 | 6.62e-03Warning: Matrix is close to singular or badly scaled. Results may be inaccurate. RCOND = 1.300371e-16.14 | 1.78e-07 | 9.63e-12 | (9.62e-12,1.00e+00) | ( 4.81e-12,-3.12e-05) | 3.12e-05 | 1.00e+0015 | 1.78e-07 | 1.93e-11 | (1.92e-11,1.00e+00) | ( 9.62e-12, 9.44e-17) | 9.62e-12 | 1.00e+0016 | 7.50e-11 | 3.35e-11 | (3.35e-11,1.00e+00) | ( 1.43e-11,-1.78e-07) | 1.78e-07 | 1.00e+00

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 245: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Example

minx∈R2

x1 + (x2 − 1)2

s.t. x ≥ 0

0 0.02 0.04 0.06 0.08 0.1

1

1.1

1.2

1.3

1.4

1.5

1.6

k | mu_k | f_k | ( x_k(1), x_k(2)) | ( p_k(1), p_k(2)) | err_barr | alpha--------------------------------------------------------------------------------------------0 | 1.00e+00 | 2.50e+00 | (2.00e+00,2.00e+00) | ( 0.00e+00, 0.00e+00) | 7.07e-01 | 0.00e+001 | 1.00e+00 | 2.02e-01 | (2.00e-02,1.60e+00) | (-2.00e+00,-4.00e-01) | 2.04e+00 | 9.90e-012 | 1.00e+00 | 2.31e-01 | (3.96e-02,1.62e+00) | ( 1.96e-02, 1.40e-02) | 2.41e-02 | 1.00e+003 | 1.00e-01 | 6.69e-02 | (6.35e-02,1.08e+00) | ( 2.39e-02,-5.36e-01) | 5.36e-01 | 1.00e+004 | 1.00e-01 | 9.09e-02 | (8.67e-02,1.09e+00) | ( 2.32e-02, 9.33e-03) | 2.50e-02 | 1.00e+005 | 1.00e-02 | 4.15e-03 | (8.67e-04,1.08e+00) | (-6.65e-01,-8.18e-02) | 6.70e-01 | 1.29e-016 | 1.00e-02 | 1.71e-03 | (1.66e-03,1.01e+00) | ( 7.92e-04,-7.12e-02) | 7.12e-02 | 1.00e+00...

12 | 3.16e-05 | 2.70e-05 | (2.70e-05,1.00e+00) | ( 7.45e-06, 2.95e-11) | 7.45e-06 | 1.00e+0013 | 1.78e-07 | 4.98e-10 | (4.81e-12,1.00e+00) | (-4.09e-03,-3.14e-05) | 4.09e-03 | 6.62e-03Warning: Matrix is close to singular or badly scaled. Results may be inaccurate. RCOND = 1.300371e-16.14 | 1.78e-07 | 9.63e-12 | (9.62e-12,1.00e+00) | ( 4.81e-12,-3.12e-05) | 3.12e-05 | 1.00e+0015 | 1.78e-07 | 1.93e-11 | (1.92e-11,1.00e+00) | ( 9.62e-12, 9.44e-17) | 9.62e-12 | 1.00e+0016 | 7.50e-11 | 3.35e-11 | (3.35e-11,1.00e+00) | ( 1.43e-11,-1.78e-07) | 1.78e-07 | 1.00e+00

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 246: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal-Dual System

minx∈Rn

f (x)

s.t. c(x) = 0x ≥ 0

∇f (x) +∇c(x)− z = 0c(x) = 0XZe = 0

(

x , z ≥ 0

)

Newton Steps∇2xxLk ∇ck −I∇cTk 0 0Zk 0 Xk

pkpλkpzk

= −

∇fk +∇ckλk − zkck

XkZke − µe

Block elimination[

∇2xxLk + X−1

k Zk ∇ck∇cTk 0

](pkλk+1

)= −

(∇fk − µX−1

k eck

)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 247: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal-Dual System

minx∈Rn

f (x)

s.t. c(x) = 0x ≥ 0

∇f (x) +∇c(x)− z = 0c(x) = 0XZe = 0

(

x , z ≥ 0

)

Newton Steps∇2xxLk ∇ck −I∇cTk 0 0Zk 0 Xk

pkpλkpzk

= −

∇fk +∇ckλk − zkck

XkZke − µe

Block elimination[

∇2xxLk + X−1

k Zk ∇ck∇cTk 0

](pkλk+1

)= −

(∇fk − µX−1

k eck

)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 248: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal-Dual System

minx∈Rn

f (x)

s.t. c(x) = 0x ≥ 0

∇f (x) +∇c(x)− z = 0c(x) = 0XZe = µe(x , z ≥ 0)

Newton Steps∇2xxLk ∇ck −I∇cTk 0 0Zk 0 Xk

pkpλkpzk

= −

∇fk +∇ckλk − zkck

XkZke − µe

Block elimination[

∇2xxLk + X−1

k Zk ∇ck∇cTk 0

](pkλk+1

)= −

(∇fk − µX−1

k eck

)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 249: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal-Dual System

minx∈Rn

f (x)

s.t. c(x) = 0x ≥ 0

∇f (x) +∇c(x)− z = 0c(x) = 0XZe = µe(x , z ≥ 0)

Newton Steps∇2xxLk ∇ck −I∇cTk 0 0Zk 0 Xk

pkpλkpzk

= −

∇fk +∇ckλk − zkck

XkZke − µe

Block elimination[∇2

xxLk + X−1k Zk ∇ck

∇cTk 0

](pkλk+1

)= −

(∇fk − µX−1

k eck

)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 250: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal-Dual System

minx∈Rn

f (x)

s.t. c(x) = 0x ≥ 0

∇f (x) +∇c(x)− z = 0c(x) = 0XZe = µe(x , z ≥ 0)

Newton Steps∇2xxLk ∇ck −I∇cTk 0 0Zk 0 Xk

pkpλkpzk

= −

∇fk +∇ckλk − zkck

XkZke − µe

Block elimination[

∇2xxLk + X−1

k Zk ∇ck∇cTk 0

](pkλk+1

)= −

(∇fk − µX−1

k eck

)

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 251: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal-Dual Steps

[∇2

xxLk + Σk ∇ck∇cTk 0

](pkλk+1

)= −

(∇fk − µX−1

k eck

)

I Barrier Hessian term:I Σk = X−2

k : primalI Σk = X−1

k Zk : primal-dual

I Step for dual variables: pzk = µX−1k e − zk − Σkpk .

I Can still use SQP-type globalization techniques.I Now: Fast local convergence.I Ill-conditioning in KKT system is benign for direct linear solvers.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 252: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal-Dual Steps

[∇2

xxLk + Σk ∇ck∇cTk 0

](pkλk+1

)= −

(∇fk − µX−1

k eck

)

I Barrier Hessian term:I Σk = X−2

k : primalI Σk = X−1

k Zk : primal-dual

I Step for dual variables: pzk = µX−1k e − zk − Σkpk .

I Can still use SQP-type globalization techniques.I Now: Fast local convergence.I Ill-conditioning in KKT system is benign for direct linear solvers.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 253: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal-Dual Steps

[∇2

xxLk + Σk ∇ck∇cTk 0

](pkλk+1

)= −

(∇fk − µX−1

k eck

)

I Barrier Hessian term:I Σk = X−2

k : primalI Σk = X−1

k Zk : primal-dual

I Step for dual variables: pzk = µX−1k e − zk − Σkpk .

I Can still use SQP-type globalization techniques.I Now: Fast local convergence.I Ill-conditioning in KKT system is benign for direct linear solvers.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 254: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal-Dual Steps

[∇2

xxLk + Σk ∇ck∇cTk 0

](pkλk+1

)= −

(∇fk − µX−1

k eck

)

I Barrier Hessian term:I Σk = X−2

k : primalI Σk = X−1

k Zk : primal-dual

I Step for dual variables: pzk = µX−1k e − zk − Σkpk .

I Can still use SQP-type globalization techniques.

I Now: Fast local convergence.I Ill-conditioning in KKT system is benign for direct linear solvers.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 255: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal-Dual Steps

[∇2

xxLk + Σk ∇ck∇cTk 0

](pkλk+1

)= −

(∇fk − µX−1

k eck

)

I Barrier Hessian term:I Σk = X−2

k : primalI Σk = X−1

k Zk : primal-dual

I Step for dual variables: pzk = µX−1k e − zk − Σkpk .

I Can still use SQP-type globalization techniques.I Now: Fast local convergence.

I Ill-conditioning in KKT system is benign for direct linear solvers.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 256: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Primal-Dual Steps

[∇2

xxLk + Σk ∇ck∇cTk 0

](pkλk+1

)= −

(∇fk − µX−1

k eck

)

I Barrier Hessian term:I Σk = X−2

k : primalI Σk = X−1

k Zk : primal-dual

I Step for dual variables: pzk = µX−1k e − zk − Σkpk .

I Can still use SQP-type globalization techniques.I Now: Fast local convergence.I Ill-conditioning in KKT system is benign for direct linear solvers.

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 257: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Example Revisited with Primal-Dual Method

minx∈R2

x1 + (x2 − 1)2

s.t. x ≥ 0

0 0.5 1 1.5 2 2.5

0

0.5

1

1.5

2

2.5

k | mu_k | f_k | ( x_k(1), x_k(2)) | ( p_k(1), p_k(2)) | err_barr | alpha--------------------------------------------------------------------------------------------0 | 1.00e+00 | 2.50e+00 | (2.00e+00,2.00e+00) | ( 0.00e+00, 0.00e+00) | 7.07e-01 | 0.00e+001 | 1.00e+00 | 2.02e-01 | (2.00e-02,1.60e+00) | (-2.00e+00,-4.00e-01) | 4.00e-03 | 9.90e-012 | 1.00e-01 | 1.22e-01 | (1.00e-01,1.21e+00) | ( 8.00e-02,-3.94e-01) | 4.74e-16 | 1.00e+003 | 1.00e-02 | 1.07e-02 | (1.00e-02,1.04e+00) | (-9.00e-02,-1.72e-01) | 5.55e-17 | 1.00e+004 | 1.00e-03 | 1.00e-03 | (1.00e-03,1.00e+00) | (-9.00e-03,-3.58e-02) | 5.55e-17 | 1.00e+005 | 3.16e-05 | 3.16e-05 | (3.16e-05,1.00e+00) | (-9.68e-04,-2.24e-03) | 3.64e-17 | 1.00e+006 | 1.78e-07 | 1.78e-07 | (1.78e-07,1.00e+00) | (-3.14e-05,-3.65e-05) | 1.10e-16 | 1.00e+007 | 7.50e-11 | 7.50e-11 | (7.50e-11,1.00e+00) | (-1.78e-07,-1.79e-07) | 5.34e-17 | 1.00e+00

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 258: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Example Revisited with Primal-Dual Method

minx∈R2

x1 + (x2 − 1)2

s.t. x ≥ 0

0.02 0.04 0.06 0.08 0.1 0.12

1

1.1

1.2

1.3

1.4

1.5

1.6

k | mu_k | f_k | ( x_k(1), x_k(2)) | ( p_k(1), p_k(2)) | err_barr | alpha--------------------------------------------------------------------------------------------0 | 1.00e+00 | 2.50e+00 | (2.00e+00,2.00e+00) | ( 0.00e+00, 0.00e+00) | 7.07e-01 | 0.00e+001 | 1.00e+00 | 2.02e-01 | (2.00e-02,1.60e+00) | (-2.00e+00,-4.00e-01) | 4.00e-03 | 9.90e-012 | 1.00e-01 | 1.22e-01 | (1.00e-01,1.21e+00) | ( 8.00e-02,-3.94e-01) | 4.74e-16 | 1.00e+003 | 1.00e-02 | 1.07e-02 | (1.00e-02,1.04e+00) | (-9.00e-02,-1.72e-01) | 5.55e-17 | 1.00e+004 | 1.00e-03 | 1.00e-03 | (1.00e-03,1.00e+00) | (-9.00e-03,-3.58e-02) | 5.55e-17 | 1.00e+005 | 3.16e-05 | 3.16e-05 | (3.16e-05,1.00e+00) | (-9.68e-04,-2.24e-03) | 3.64e-17 | 1.00e+006 | 1.78e-07 | 1.78e-07 | (1.78e-07,1.00e+00) | (-3.14e-05,-3.65e-05) | 1.10e-16 | 1.00e+007 | 7.50e-11 | 7.50e-11 | (7.50e-11,1.00e+00) | (-1.78e-07,-1.79e-07) | 5.34e-17 | 1.00e+00

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 259: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Table of Contents

Applications

Equality-Constrained Quadratic Programming

Active-Set Quadratic Programming Solvers

SQP for Equality-Constrained NLPs

SQP for Inequality-Constrained NLPs

Interior Point Methods

Software

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 260: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Advantages of Different Algorithms

(very rough guide. . . )

SQP methodsI Very efficient for small- to medium-sized problems

I up to several thousand variables and constraintsI Can exploit good estimate of solution (warm starts)

I branch-and-bound for mixed-integer nonlinear programmingI real-time optimal control

Interior-point methodsI Can solve very large problems

I up to millions of variables and constraintsI Difficult to warm start

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 261: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Advantages of Different Algorithms

(very rough guide. . . )

SQP methodsI Very efficient for small- to medium-sized problems

I up to several thousand variables and constraints

I Can exploit good estimate of solution (warm starts)I branch-and-bound for mixed-integer nonlinear programmingI real-time optimal control

Interior-point methodsI Can solve very large problems

I up to millions of variables and constraintsI Difficult to warm start

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 262: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Advantages of Different Algorithms

(very rough guide. . . )

SQP methodsI Very efficient for small- to medium-sized problems

I up to several thousand variables and constraintsI Can exploit good estimate of solution (warm starts)

I branch-and-bound for mixed-integer nonlinear programmingI real-time optimal control

Interior-point methodsI Can solve very large problems

I up to millions of variables and constraintsI Difficult to warm start

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 263: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Advantages of Different Algorithms

(very rough guide. . . )

SQP methodsI Very efficient for small- to medium-sized problems

I up to several thousand variables and constraintsI Can exploit good estimate of solution (warm starts)

I branch-and-bound for mixed-integer nonlinear programmingI real-time optimal control

Interior-point methodsI Can solve very large problems

I up to millions of variables and constraints

I Difficult to warm start

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 264: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Advantages of Different Algorithms

(very rough guide. . . )

SQP methodsI Very efficient for small- to medium-sized problems

I up to several thousand variables and constraintsI Can exploit good estimate of solution (warm starts)

I branch-and-bound for mixed-integer nonlinear programmingI real-time optimal control

Interior-point methodsI Can solve very large problems

I up to millions of variables and constraintsI Difficult to warm start

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 265: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Some Optimization Software for NLP

SQP methods

(This is not an exhaustive list!)

I SNOPT [Gill, Murray, Sanders]I line search with augmented Lagrangian as merit functionI reduced Hessian BFGS

I FilterSQP [Fletcher, Leyffer]I S`1QP with exact HessianI trust-region filter method

Primal-dual interior-point methodsI Ipopt [Wächter, Biegler]

I line-search filter methodI full-space step computation with regularization

I Knitro [Byrd, Nocedal, Waltz et al.]I trust-region with exact penalty functionI Byrd-Omojokun decompositionI other algorithmic options: direct method; SLP-EQP method

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 266: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Some Optimization Software for NLPSQP methods (This is not an exhaustive list!)

I SNOPT [Gill, Murray, Sanders]I line search with augmented Lagrangian as merit functionI reduced Hessian BFGS

I FilterSQP [Fletcher, Leyffer]I S`1QP with exact HessianI trust-region filter method

Primal-dual interior-point methodsI Ipopt [Wächter, Biegler]

I line-search filter methodI full-space step computation with regularization

I Knitro [Byrd, Nocedal, Waltz et al.]I trust-region with exact penalty functionI Byrd-Omojokun decompositionI other algorithmic options: direct method; SLP-EQP method

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 267: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Some Optimization Software for NLPSQP methods (This is not an exhaustive list!)

I SNOPT [Gill, Murray, Sanders]I line search with augmented Lagrangian as merit functionI reduced Hessian BFGS

I FilterSQP [Fletcher, Leyffer]I S`1QP with exact HessianI trust-region filter method

Primal-dual interior-point methodsI Ipopt [Wächter, Biegler]

I line-search filter methodI full-space step computation with regularization

I Knitro [Byrd, Nocedal, Waltz et al.]I trust-region with exact penalty functionI Byrd-Omojokun decompositionI other algorithmic options: direct method; SLP-EQP method

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 268: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Some Optimization Software for NLPSQP methods (This is not an exhaustive list!)

I SNOPT [Gill, Murray, Sanders]I line search with augmented Lagrangian as merit functionI reduced Hessian BFGS

I FilterSQP [Fletcher, Leyffer]I S`1QP with exact HessianI trust-region filter method

Primal-dual interior-point methodsI Ipopt [Wächter, Biegler]

I line-search filter methodI full-space step computation with regularization

I Knitro [Byrd, Nocedal, Waltz et al.]I trust-region with exact penalty functionI Byrd-Omojokun decompositionI other algorithmic options: direct method; SLP-EQP method

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 269: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Some Optimization Software for NLPSQP methods (This is not an exhaustive list!)

I SNOPT [Gill, Murray, Sanders]I line search with augmented Lagrangian as merit functionI reduced Hessian BFGS

I FilterSQP [Fletcher, Leyffer]I S`1QP with exact HessianI trust-region filter method

Primal-dual interior-point methodsI Ipopt [Wächter, Biegler]

I line-search filter methodI full-space step computation with regularization

I Knitro [Byrd, Nocedal, Waltz et al.]I trust-region with exact penalty functionI Byrd-Omojokun decompositionI other algorithmic options: direct method; SLP-EQP method

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 270: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Some Optimization Software for NLP

Augmented Lagrangian methodsI Lancelot [Conn, Gould, Toint]

I trust regionI gradient projection combined with conjugate gradients

I Algencan [Birgin, Martinez]I trust regionI spectral projected gradients

Based on others algorithmic frameworksI CONOPT [Arki Consulting]

I “based on generalize reduced-gradient method”I MINOS [Murtagh, Sanders]

I linearly-constrained augmented Lagrangian methodI line search

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 271: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Some Optimization Software for NLP

Augmented Lagrangian methodsI Lancelot [Conn, Gould, Toint]

I trust regionI gradient projection combined with conjugate gradients

I Algencan [Birgin, Martinez]I trust regionI spectral projected gradients

Based on others algorithmic frameworksI CONOPT [Arki Consulting]

I “based on generalize reduced-gradient method”I MINOS [Murtagh, Sanders]

I linearly-constrained augmented Lagrangian methodI line search

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 272: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Some Optimization Software for NLP

Augmented Lagrangian methodsI Lancelot [Conn, Gould, Toint]

I trust regionI gradient projection combined with conjugate gradients

I Algencan [Birgin, Martinez]I trust regionI spectral projected gradients

Based on others algorithmic frameworksI CONOPT [Arki Consulting]

I “based on generalize reduced-gradient method”

I MINOS [Murtagh, Sanders]I linearly-constrained augmented Lagrangian methodI line search

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 273: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Some Optimization Software for NLP

Augmented Lagrangian methodsI Lancelot [Conn, Gould, Toint]

I trust regionI gradient projection combined with conjugate gradients

I Algencan [Birgin, Martinez]I trust regionI spectral projected gradients

Based on others algorithmic frameworksI CONOPT [Arki Consulting]

I “based on generalize reduced-gradient method”I MINOS [Murtagh, Sanders]

I linearly-constrained augmented Lagrangian methodI line search

Andreas Wächter Constrained Nonlinear Optimization Algorithms

Page 274: Constrained Nonlinear Optimization Algorithms · g b!. Andreas Wächter Constrained Nonlinear Optimization Algorithms. Equality-ConstrainedQP min x∈Rn 1 2 x TQx+ gTx s.t.Ax+ b=

Thank You!

Andreas Wächter Constrained Nonlinear Optimization Algorithms