General Nonlinear Programming (NLP) Softwarecs777/presentations/NLPSoftware.pdf · General...

General Nonlinear Programming (NLP) Software

CAS 737 / CES 735

Kristin DaviesHamid Ghaffari

Alberto Olvera-SalazarVoicu Chis

January 12, 2006

OutlineIntro to NLPExamination of:

IPOPT PENNON CONOPT LOQO KNITRO

Comparison of Computational ResultsConclusions

Intro to NLPThe general problem:

{ }{ }

.

,,,

,...,1,0)(,...,1,0)(..

)(min

Condefinedfunctionsare

ghfandsetcertainaisCxwhereCx

mJjxgpIixhts

xf

jinn

j

i

ℜ⊆ℜ∈

∈

=∈≤=∈=

Either the objective function or some of the constraints may be nonlinear

(NLP)

Intro to NLP (cont’d…)Recall:

The feasible region of any LP is a convex setif the LP has an optimal solution, there is an extreme point of the feasible set that is optimal

However:even if the feasible region of an NLP is a convex set, the optimal solution might not be an extreme point of the feasible region

Intro to NLP (cont’d…)Some Major approaches for NLP

Interior Point MethodsUse a log-barrier function

Penalty and Augmented Lagrange MethodsUse the idea of penalty to transform a constrained problem into a sequence of unconstrained problems.

Generalized reduced gradient (GRG)Use a basic Descent algorithm.

Successive quadratic programming (SQP)Solves a quadratic approximation at every iteration.

Summary of NLP Solvers

PENNON

AugmentedLagrangian Methods

KNITRO (TR)IPOPT, LOQO(line search)

Interior PointMethods

CONOPT

Reduced GradientMethods

NLP

IPOPT SOLVER (Interior Point OPTimizer)

CreatorsAndreas Wachter and L.T. Biegler at CMU (~2002)

AimsSolver for Large-Scale Nonlinear Optimization problems

ApplicationsGeneral Nonlinear optimizationProcess Engineering, DAE/PDE Systems, Process Design and Operations, Nonlinear Model Predictive control, Design Under Uncertainty


Input FormatCan be linked to Fortran and C code MATLAB and AMPL.

Language / OSFortran 77, C++ (Recent Version IPOPT 3.x)Linux/UNIX platforms and Windows

Commercial/FreeReleased as open source code under the Common Public License (CPL). It is available from the COIN-OR repository

http://www.opensource.org/licenses/cpl.php




Key ClaimsGlobal Convergence by using a Line Search.

Find a KKT pointPoint that Minimizes Infeasibility (locally)

Exploits Exact Second DerivativesAMPL (automatic differentiation)If not Available use QN approx (BFGS)Sparsity of the KKT matrix.

IPOPT has a version to solve problems with MPEC Constraints. (IPOPT-C)


AlgorithmInterior Point method with a novel line search filter.Optimization Problem

00)(..

)(min

≥=

ℜ∈

xxcts

xfnx

0)(..

)log()()(min )(

=

−= ∑ℜ∈

xcts

xxfxi

il

xln

μϕμ

The bounds are replaced by a logarithmic Barrier term. The method solves a sequence of barrier problems for decreasing values of μl


Algorithm(For a fixed value of )

Solve the Barrier ProblemSearch Direction (Primal-Dual IP)

Use a Newton method to solve the primal dual equations.Hessian Approximation (BFGS update)

Line Search (Filter Method)Feasibility Restoration Phase

lμ


Optimization Problem

00)(..

)(min

≥=

ℜ∈

xxcts

xfnx

0)(..

)log()()(min )(

=

−= ∑ℜ∈

xcts

xxfxi

il

xln

μϕμ

lμ

Outer LoopOuter Loop

The bounds are replaced by a logarithmic Barrier term.

The method solves a sequence of barrier problems for decreasing values of


Algorithm (For a fixed value of )Solve the Barrier Problem

Search Direction (Primal-Dual IP)Use a Newton method to solve the primal dual equationsHessian Approximation (BFGS update)

lμ


InnerLoopInnerLoop

0)(0

0)()(

==−

=−∇+∇

xceXVe

vxcxfμ

λ

Optimality conditionsOptimality conditions

eXvVariablesDual

1−= μ

0)(..

)log()()(min )(

=

−= ∑ℜ∈

xcts

xxfxi

il

xln

μϕμ Barrier Barrier NLPNLP

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−

−∇+∇−=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡∇

−∇

eeVXxc

vxcxf

ddd

VXxc

IxcH

kk

k

kkT

kk

vk

k

xk

kk

Tk

kk

μ

λλ )(

)()(

000)(

)(

),(2kkxxk xLH λ∇=

At a Newton's iteration At a Newton's iteration ((xxkk,,λλkk,v,vkk))

⎥⎦

⎤⎢⎣

⎡ −∇+∇−=⎥

⎦

⎤⎢⎣

⎡⎥⎦

⎤⎢⎣

⎡−∇∇+Σ+

)()()(

)()(

k

kkT

kk

k

xk

cT

k

kHkk

xcvxcx

dd

IxcxcIH λϕδ

δ μλ

At a Newton's iteration At a Newton's iteration ((xxkk,,λλkk,v,vkk))

Algorithm Core: Solution of this Linear systemAlgorithm Core: Solution of this Linear system


Algorithm (For a fixed value of )Line Search (Filter Method)

A trial point is accepted if improves feasibility or if improves the barrier function

Assumes Newton directions are “Good”especially when using Exact 2nd Derivatives

lμ

vkkkk

xkkkk

dvv

dxx

α

α

+=

+=

+

+

1

1

[ ] [ ]

[ ] [ ]kkk

kkk

xxor

xcxc

ϕαϕ

α

≤

≤

)(

)(If


Line Search - Feasibility Restoration Phase

When a new trial point does not provides sufficient improvement.

0..

)(min 2

2

≥ℜ∈

xts

xcnx

Restore FeasibilityMinimize constraint

violation

00)(..

min 2

2

≥=

−ℜ∈

xxcts

xx kx n

Force Unique SolutionFind closest feasible point.

Add Penalty function

The complexity of the problem increases when complementarity conditions are introduced from:

miywywx

ywxcst

ywxf

ii

ywx mmn

K1,00,,

0),,(.

),,(min

)()(

,,

==

≥=

ℜ∈ℜ∈ℜ∈

( ) ( ) ( )( )

miywywxc

st

ywx

ywxf

ii

m

iim

iin

ii

ywx mmn

K1,00),,(

.

lnlnln

),,(min

)()(

1)(

1)(

1)(

,,

==

=

++− ∑∑∑ ===

ℜ∈ℜ∈ℜ∈

μ

δμ≤)()( ii yw

•The interior Point method for NLPs has been extended to handle complementarity problems. (Raghunathan et al. 2003).

0)()( =ii yw is relaxed as 0)(

)()()(

≥

=+i

iii

ssyw δμ



AdditionalIPOPT 3x. Is now programmed in C++.Is the primary NLP Solver in an undergoing project for MINLP with IBM.

ReferencesIpopt homepage:

http://www.coin-or.org/Ipopt/ipopt-fortran.htmlA. Wächter and L. T. Biegler, On the Implementation of a Primal-Dual Interior Point Filter Line Search Algorithm for Large-Scale Nonlinear Programming, Research Report, IBM T. J. Watson Research Center, Yorktown, USA, (March 2004 - accepted for publication in Mathematical Programming)

http://www.coin-or.org/Ipopt/ipopt-fortran.html

PENNON (PENalty method for NONlinear& semidefinite programming)

CreatorsMichal Kocvara & Michael Stingl (~2001)

AimsNLP, Semidefinite Programming (SDP), Linear & Bilinear Matrix Inequalities (LMI & BMI), Second Order Conic Programming (SOCP)

ApplicationsGeneral purpose nonlinear optimization, systems of equations, control theory, economics & finance, structural optimization, engineering

SDP (SemiDefinite Programming)

Minimization of a linear function subject to the constraint that an affine combination of symmetric matrices is positive semidefinite

∑=

+=

≥m

iii

T

FxFxFwhere

xFtsxc

10)(

0)(..min

),...,(matricessymmetric1 0 mFFm +

Linear Matrix Inequality (LMI)defines a convex constraint on x

SDP (SemiDefinite Programming)

-always an optimal point on the boundary

-boundary consists of piecewise algebraic surfaces

SOCP (Second-Order Conic Programming)

Minimization of a linear function subject to a second-order cone constraint

iTiii

T

dxcbxAtsxc

+≤+..min

⎭⎬⎫

⎩⎨⎧

≤∈∈⎥⎦

⎤⎢⎣

⎡= − tuRtRu

tu

C kk ,,1

Called a second-order cone constraint since the unit second-order cone of dimension k is defined as:

Which is called thequadratic, ice-cream,or Lorentz cone


Input FormatMATLAB function, routine called from C or Fortran, stand-alone program with AMPL

LanguageFortran 77

Commercial/FreeVariety of licenses ranging from

Academic – single user ($460 CDN) to Commercial – company ($40,500 CDN)


Key Claims1st available code for combo NLP, LMI, & BMI constraintsAimed at (very) large-scale problemsEfficient treatment of different sparsitypatterns in problem dataRobust with respect to feasibility of initial guessParticularly efficient for large convex problems


AlgorithmGeneralized version of the Augmented Langrangian method (originally by Ben-Tal & Zibulevsky)

( ) 0/)(..)(min

≤iigi pxgptsxfϕ

Augmented Lagrangian

( )∑=

+=gm

iiigii pxgpuxfpuxF

1/)()(),,( ϕ

multiplierLagrange

functionpenaltyparameterpenalty0

sconstraintinequalityof#

,...,1

=

==>

=Augmented Problem =

i

g

i

g

g

u

p

m

mi

ϕ


The AlgorithmConsider only inequality constraints from (NLP)

Based on choice of a penalty function, φg, that penalizes the inequality constraints

Penalty function must satisfy multiple properties such that the original (NLP) has the same solution as the following “augmented” problem:

( )0

,...1,0/)(..,)(min

>

=≤ℜ∈

i

giigi

pwith

mipxgptsxxf

ϕ (NLPφ)

[3] Kocvara & Stingl


The Algorithm (Cont’d…)The Lagrangian of (NLPφ) can be viewed as a (generalized) augmented Lagrangian of (NLP):

( )∑=

+=gm

iiigii pxgpuxfpuxF

1/)()(),,( ϕ

Lagrange multiplierPenalty parameter

Penalty function

Inequality constraint



The Algorithm STEPS.,...,1,0.)0( 111

gii mipLetgivenbeuandxLet =>

.,...2,1

satisfiediscriteriumstoppingauntilrepeatkFor =


( )( )( )

gki

ki

gki

kig

ki

ki

kkkx

k

mippiii

mipxguuii

KpuxFthatsuchxFindi

,...,1,)(

,...,1,/)(

,,)(

1

1'1

11

=<

==

≤∇

+

++

++

ϕ


The Algorithm STEPS

.,...,1,0.)0( 111gii mipLetgivenbeuandxLet =>

InitializationCan start with an arbitrary primal variable , therefore, chooseCalculate initial multiplier valuesInitial p= , typically between 10 - 10000

1iu

01 =xx


0π


The Algorithm STEPS

( ) KpuxFthatsuchxFindi kkkx

k ≤∇ ++ ,,)( 11

(Approximate) Unconstrained MinimizationPerformed either by Newton with Line Search, or by Trust RegionStopping criteria:


( )( ) ( )( )( ) ( ) 11 ,,,,

/,,

1.0,,

1

2

1'

2

1

2

1

−−∇⋅≤∇

−⋅≤∇

=≤∇

+

++

+

H

kkkxH

kkkx

ki

kig

ki

ki

kkkx

kkkx

puxFpuxF

orpxguupuxF

orpuxF

α

ψα

αα

1)'(' −= ϕψ),,(minarg1 kk

x

k puxFx =+


The Algorithm STEPS

5.0,1 typically≤μ

( )( ) gki

kig

ki

ki mipxguuii ,...,1,/)( 1'1 == ++ ϕ

Update of MultipliersRestricted in order to satisfy:

with a positive

If left-side violated, let If right side violate, let

μμ 11

<<+

ki

ki

uu

μ=newiu

μ/1=newiu



The Algorithm STEPS

gki

ki mippiii ,...,1,)( 1 =<+

Update of Penalty ParameterNo update during first 3 iterationsAfterwards, updated by a constant factor dependent on initial penalty parameterPenalty update is stopped if peps (10-6) is reached



The Algorithm

Choice of Penalty FunctionMost efficient penalty function for convex NLP is the quadratic-logarithmic function:

[4] Ben-Tal & Zibulevsky

holdpropertiesthatsocrtcctc

andrwherertctctct

i

g

6,...,1)log(

)1,1()(

654

322

21

1

=<+−

−∈≥++=ϕ


The Algorithm

Overall Stopping Criteria


7

1

10

)(1

)()(

)(1

),,()(

−

−

=

<+

−<

+

−

ε

εε

where

xf

xfxfor

xf

puxFxfk

kk

k

kkk


Assumptions / WarningsMore tuning for nonconvex problems is still requiredSlower at solving linear SDP problems since algorithm is generalized


ReferencesKocvara, Michal & Michael Stingl. PENNON: A Code for Convex and Semidefinite Programming. Optimization Methods and Software, 8(3):317-333, 2003.Kocvara, Michal & Michael Stingl. PENNON-AMPL User’s Guide. www.penopt.com . August 2003.Ben-Tal, Aharon & Michael Zibulevsky. Penalty/Barrier Multiplier Methods for Convex Programming Problems. Siam J. Optim., 7(2):347-366, 1997.Pennon Homepage. www.penopt.com/pennon.html Available online January 2007.

http://www.penopt.com/

http://www.penopt.com/pennon.html

General Nonlinear Programming (NLP) Softwarecs777/presentations/NLPSoftware.pdf · General...

Documents

Transcript of General Nonlinear Programming (NLP) Softwarecs777/presentations/NLPSoftware.pdf · General...