Mixed Integer Nonlinear Programming - Lehigh...

90
Tutorial: Mixed Integer Nonlinear Programming (MINLP) Sven Leyffer MCS Division Argonne National Lab [email protected] Jeff Linderoth ISE Department Lehigh University [email protected] informs Annual Meeting San Francisco May 15, 2005 Leyffer & Linderoth MINLP

Transcript of Mixed Integer Nonlinear Programming - Lehigh...

Page 1: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Tutorial:

Mixed Integer Nonlinear Programming (MINLP)

Sven Leyffer

MCS DivisionArgonne National [email protected]

Jeff Linderoth

ISE DepartmentLehigh [email protected]

informs Annual MeetingSan FranciscoMay 15, 2005

Leyffer & Linderoth MINLP

Page 2: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

New Math

MI

+

NLP

6=

MINLP

Leyffer & Linderoth MINLP

Page 3: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Tutorial Overview

1. Introduction, Applications, and Formulations

2. Classical Solution Methods

3. Modern Developments in MINLP

4. Implementation and Software

Leyffer & Linderoth MINLP

Page 4: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

MotivationExamples

Tricks

Part I

Introduction, Applications, and Formulations

Leyffer & Linderoth MINLP

Page 5: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

MotivationExamples

Tricks

WhatHowWhy?

The Problem of the Day

Mixed Integer Nonlinear Program (MINLP)minimize

x,yf(x, y)

subject to c(x, y) ≤ 0x ∈ X, y ∈ Y integer

• f, c smooth (convex) functions

• X,Y polyhedral sets, e.g. Y = y ∈ [0, 1]p | Ay ≤ b• y ∈ Y integer ⇒ hard problem

• f, c not convex ⇒ very hard problem

Leyffer & Linderoth MINLP

Page 6: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

MotivationExamples

Tricks

WhatHowWhy?

Why the N?

An anecdote: July, 1948. A young andfrightened George Dantzig, presents hisnewfangled “linear programming” to ameeting of the Econometric Society ofWisconsin, attended by distinguishedscientists like Hotelling, Koopmans, andVon Neumann. Following the lecture,Hotellinga pronounced to the audience:

But we all know the world isnonlinear!

ain Dantzig’s words “a huge whale of aman”

The world is indeednonlinear

• Physical Processesand Properties

• Equilibrium• Enthalpy

• Abstract Measures• Economies of

Scale• Covariance• Utility of decisions

Leyffer & Linderoth MINLP

Page 7: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

MotivationExamples

Tricks

WhatHowWhy?

Why the MI?

• We can use 0-1 (binary) variables for a variety of purposes• Modeling yes/no decisions• Enforcing disjunctions• Enforcing logical conditions• Modeling fixed costs• Modeling piecewise linear functions

• If the variable is associated with a physical entity that isindivisible, then it must be integer

• Number of aircraft carriers to to produce. Gomory’s InitialMotivation

Leyffer & Linderoth MINLP

Page 8: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

MotivationExamples

Tricks

WhatHowWhy?

A Popular MINLP Method

Dantzig’s Two-Phase Method for MINLP Adapted by Leyffer and Linderoth

1. Convince the user that he or she does not wish to solve amixed integer nonlinear programming problem at all!

2. Otherwise, solve the continuous relaxation (NLP ) and roundoff the minimizer to the nearest integer.

• For 0− 1 problems, or those in which the |y| is “small”, thecontinuous approximation to the discrete decision is notaccurate enough for practical purposes.

• Conclusion: MINLP methods must be studied!

Leyffer & Linderoth MINLP

Page 9: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

MotivationExamples

Tricks

WhatHowWhy?

Example: Core Reload Operation (Quist, A.J., 2000)

• max. reactor efficiency after reloadsubject to diffusion PDE & safety

• diffusion PDE ' nonlinear equation⇒ integer & nonlinear model

• avoid reactor becoming overheated

Leyffer & Linderoth MINLP

Page 10: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

MotivationExamples

Tricks

WhatHowWhy?

Example: Core Reload Operation (Quist, A.J., 2000)

• look for cycles for moving bundles:e.g. 4 → 6 → 8 → 10i.e. bundle moved from 4 to 6 ...

• model with binary xilm ∈ 0, 1xilm = 1⇔ node i has bundle l of cycle m

Leyffer & Linderoth MINLP

Page 11: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

MotivationExamples

Tricks

WhatHowWhy?

AMPL Model of Core Reload Operation

Exactly one bundle per node:

L∑l=1

M∑m=1

xilm = 1 ∀i ∈ I

AMPL model:var x I,L,M binary ;Bundle i in I: suml in L, m in M x[i,l,m] = 1 ;

• Multiple Choice: One of the most common uses of IP

• Full AMPL model c-reload.mod atwww.mcs.anl.gov/∼leyffer/MacMINLP/

Leyffer & Linderoth MINLP

Page 12: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

MotivationExamples

Tricks

Gas TransmissionPortfolio ManagementBatch Processing

Gas Transmission Problem (De Wolf and Smeers, 2000)

• Belgium has no gas!

• All natural gas isimported from Norway,Holland, or Algeria.

• Supply gas to all demandpoints in a network in aminimum cost fashion.

• Gas is pumped throughthe network with a seriesof compressors

• There are constraints onthe pressure of the gaswithin the pipe

Leyffer & Linderoth MINLP

Page 13: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

MotivationExamples

Tricks

Gas TransmissionPortfolio ManagementBatch Processing

Pressure Loss is Nonlinear

• Assume horizontal pipes andsteady state flows

• Pressure loss p across a pipeis related to the flow rate fas

p2in − p2

out =1

Ψsign(f)f2

• Ψ: “Friction Factor”

Leyffer & Linderoth MINLP

Page 14: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

MotivationExamples

Tricks

Gas TransmissionPortfolio ManagementBatch Processing

Gas Transmission: Problem Input

• Network (N,A). A = Ap ∪Aa

• Aa: active arcs have compressor. Flow rate can increase on arc• Ap: passive arcs simply conserve flow rate

• Ns ⊆ N : set of supply nodes

• ci, i ∈ Ns: Purchase cost of gas

• si, si: Lower and upper bounds on gas “supply” at node i

• pi, pi: Lower and upper bounds on gas pressure at node i

• si, i ∈ N : supply at node i.• si > 0⇒ gas added to the network at node i• si < 0⇒ gas removed from the network at node i to meet

demand

• fij , (i, j) ∈ A: flow along arc (i, j)• f(i, j) > 0⇒ gas flows i→ j• f(i, j) < 0⇒ gas flows j → i

Leyffer & Linderoth MINLP

Page 15: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

MotivationExamples

Tricks

Gas TransmissionPortfolio ManagementBatch Processing

Gas Transmission Model

min∑j∈Ns

cjsj

subject to ∑j|(i,j)∈A

fij = si ∀i ∈ N

sign(fij)f2ij −Ψij(p

2i − p2

j ) = 0 ∀(i, j) ∈ Ap

sign(fij)f2ij −Ψij(p

2i − p2

j ) ≥ 0 ∀(i, j) ∈ Aa

si ∈ [si, si] ∀i ∈ Npi ∈ [p

i, pi] ∀i ∈ N

fij ≥ 0 ∀(i, j) ∈ Aa

Leyffer & Linderoth MINLP

Page 16: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

MotivationExamples

Tricks

Gas TransmissionPortfolio ManagementBatch Processing

Your First Modeling Trick

• Don’t include nonlinearities or nonconvexities unless necessary!

• Replace p2i ← ρi

sign(fij)f2ij −Ψij(ρi − ρj) = 0 ∀(i, j) ∈ Ap

f2ij −Ψij(ρi − ρj) ≥ 0 ∀(i, j) ∈ Aa

ρi ∈ [√p

i,√pi] ∀i ∈ N

• This trick only works because

1. p2i terms appear only in the bound constraints

2. Also fij ≥ 0 ∀(i, j) ∈ Aa

• This model is nonconvex: sign(fij)f2ij is a nonconvex function

• Some solvers do not like sign

Leyffer & Linderoth MINLP

Page 17: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

MotivationExamples

Tricks

Gas TransmissionPortfolio ManagementBatch Processing

Dealing with sign(·): The NLP Way

• Use auxiliary binary variables to indicate direction of flow

• Let |fij | ≤ F ∀(i, j) ∈ Ap

zij =

1 fij ≥ 0 fij ≥ −F (1− zij)0 fij ≤ 0 fij ≤ Fzij

• Note thatsign(fij) = 2zij − 1

• Write constraint as

(2zij − 1)f2ij −Ψij(ρi − ρj) = 0.

Leyffer & Linderoth MINLP

Page 18: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

MotivationExamples

Tricks

Gas TransmissionPortfolio ManagementBatch Processing

Special Ordered Sets

• Sven thinks this ’NLP trick’ is pretty cool

• It is not how it is done in De Wolf and Smeers (2000).

• Heuristic for finding a good starting solution, then a localoptimization approach based on a piecewise-linear simplexmethod

• Another (similar) approach involves approximating thenonlinear function by piecewise linear segments, but searchingfor the globally optimal solution: Special Ordered Sets ofType 2

• If the “multidimensional” nonlinearity cannot be removed,resort to Special Ordered Sets of Type 3

Leyffer & Linderoth MINLP

Page 19: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

MotivationExamples

Tricks

Gas TransmissionPortfolio ManagementBatch Processing

Portfolio Management

• N : Universe of asset to purchase

• xi: Amount of asset i to hold

• B: Budget

minx∈R|N|

+

u(x) |

∑i∈N

xi = B

• Markowitz: u(x)def= −αTx+ λxTQx

• α: Expected returns• Q: Variance-covariance matrix of expected returns• λ: Risk aversion parameter

Leyffer & Linderoth MINLP

Page 20: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

MotivationExamples

Tricks

Gas TransmissionPortfolio ManagementBatch Processing

More Realistic Models

• b ∈ R|N | of “benchmark” holdings

• Benchmark Tracking: u(x)def= (x− b)TQ(x− b)

• Constraint on E[Return]: αTx ≥ r• Limit Names: |i ∈ N : xi > 0| ≤ K

• Use binary indicator variables to model the implicationxi > 0⇒ yi = 1

• Implication modeled with variable upper bounds:

xi ≤ Byi ∀i ∈ N

•∑

i∈N yi ≤ K

Leyffer & Linderoth MINLP

Page 21: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

MotivationExamples

Tricks

Gas TransmissionPortfolio ManagementBatch Processing

Even More Models

• Min Holdings: (xi = 0) ∨ (xi ≥ m)• Model implication: xi > 0⇒ xi ≥ m• xi > 0⇒ yi = 1⇒ xi ≥ m• xi ≤ Byi, xi ≥ myi ∀i ∈ N

• Round Lots: xi ∈ kLi, k = 1, 2, . . .• xi − ziLi = 0, zi ∈ Z+ ∀i ∈ N

• Vector h of initial holdings

• Transactions: ti = |xi − hi|• Turnover:

∑i∈N ti ≤ ∆

• Transaction Costs:∑

i∈N citi in objective

• Market Impact:∑

i∈N γit2i in objective

Leyffer & Linderoth MINLP

Page 22: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

MotivationExamples

Tricks

Gas TransmissionPortfolio ManagementBatch Processing

Multiproduct Batch Plants (Kocis and

Grossmann, 1988)

• M : Batch Processing Stages

• N : Different Products

• H: Horizon Time

• Qi: Required quantity of product i

• tij : Processing time product i stage j

• Sij : “Size Factor” product i stage j

• Bi: Batch size of product i ∈ N• Vj : Stage j size: Vj ≥ SijBi ∀i, j• Nj : Number of machines at stage j

• Ci: Longest stage time for product i: Ci ≥ tij/Nj ∀i, jLeyffer & Linderoth MINLP

Page 23: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

MotivationExamples

Tricks

Gas TransmissionPortfolio ManagementBatch Processing

Multiproduct Batch Plants

min∑j∈M

αjNjVβj

j

s.t.

Vj − SijBi ≥ 0 ∀i ∈ N,∀j ∈MCiNj ≥ tij ∀i ∈ N,∀j ∈M∑

i∈N

Qi

BiCi ≤ H

Bound Constraints on Vj , Ci, Bi, Nj

Nj ∈ Z ∀j ∈M

Leyffer & Linderoth MINLP

Page 24: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

MotivationExamples

TricksVariable Transformation

Modeling Trick #2

• Horizon Time and Objective Function Nonconvex. :-(

• Sometimes variable transformations work!

vj = ln(Vj), nj = ln(Nj), bi = ln(Bi), ci = lnCi

min∑j∈M

αjeNj+βjVj

s.t. vj − ln(Sij)bi ≥ 0 ∀i ∈ N,∀j ∈Mci + nj ≥ ln(τij) ∀i ∈ N,∀j ∈M∑

i∈N

QieCi−Bi ≤ H

(Transformed) Bound Constraints on Vj , Ci, Bi

Leyffer & Linderoth MINLP

Page 25: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

MotivationExamples

TricksVariable Transformation

How to Handle the Integrality?

• But what to do about the integrality?

1 ≤ Nj ≤ N j ∀j ∈M,Nj ∈ Z ∀j ∈M

• nj ∈ 0, ln(2), ln(3), . . . ...

Ykj =

1 nj takes value ln(k)0 Otherwise

nj −K∑

k=1

ln(k)Ykj = 0 ∀j ∈M

K∑k=1

Ykj = 1 ∀j ∈M

• This model is available at http://www-unix.mcs.anl.gov/∼leyffer/macminlp/problems/batch.mod

Leyffer & Linderoth MINLP

Page 26: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

MotivationExamples

TricksVariable Transformation

A Small Smattering of Other Applications

• Chemical Engineering Applications:• process synthesis (Kocis and Grossmann, 1988)• batch plant design (Grossmann and Sargent, 1979)• cyclic scheduling (Jain, V. and Grossmann, I.E., 1998)• design of distillation columns (Viswanathan and Grossmann,

1993)• pump configuration optimization (Westerlund, T., Pettersson,

F. and Grossmann, I.E., 1994)

• Forestry/Paper• production (Westerlund, T., Isaksson, J. and Harjunkoski, I.,

1995)• trimloss minimization (Harjunkoski, I., Westerlund, T., Porn,

R. and Skrifvars, H., 1998)

• Topology Optimization (Sigmund, 2001)

Leyffer & Linderoth MINLP

Page 27: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Branch-and-BoundOuter Approximation

Hybrid Methods

Part II

Classical Solution Methods

Leyffer & Linderoth MINLP

Page 28: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Branch-and-BoundOuter Approximation

Hybrid Methods

Classical Solution Methods for MINLP

1. Classical Branch-and-Bound

2. Outer Approximation & Benders Decomposition

3. Hybrid Methods• LP/NLP Based Branch-and-Bound• Integrating SQP with Branch-and-Bound

Leyffer & Linderoth MINLP

Page 29: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Branch-and-BoundOuter Approximation

Hybrid MethodsDefinition

Branch-and-Bound

Solve relaxed NLP (0 ≤ y ≤ 1 continuous relaxation). . . solution value provides lower bound

• Branch on yi non-integral

• Solve NLPs & branch until

1. Node infeasible ... •2. Node integer feasible ... ⇒ get upper bound (U)

3. Lower bound ≥ U ...⊗

y = 1

y = 0i

i

dominated by upper bound

infeasible

integer feasibleetc.

etc.

Search until no unexplored nodes on tree

Leyffer & Linderoth MINLP

Page 30: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Branch-and-BoundOuter Approximation

Hybrid MethodsDefinition

Variable Selection for Branch-and-BoundAssume yi ∈ 0, 1 for simplicity ...(x, y) fractional solution to parent node; f = f(x, y)

1. maximal fractional branching: choose yi closest to 12

maximin(1− yi, yi)

2. strong branching: (approx) solve all NLP children:

f+/−i ←

minimize

x,yf(x, y)

subject to c(x, y) ≤ 0x ∈ X, y ∈ Y, yi = 1/0

branching variable yi that changes objective the most:

maxi

min(f+

i , f−i )

Leyffer & Linderoth MINLP

Page 31: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Branch-and-BoundOuter Approximation

Hybrid MethodsDefinition

Node Selection for Branch-and-Bound

Which node n on tree T should be solved next?

1. depth-first search: select deepest node in tree• minimizes number of NLP nodes stored• exploit warm-starts (MILP/MIQP only)

2. best estimate: choose node with best expected integer soln

minn∈T

fp(n) +∑

i:yifractional

mine+i (1− yi), e

−i yi

where fp(n) = value of parent node, e

+/−i = pseudo-costs

summing pseudo-cost estimates for all integers in subtree

Leyffer & Linderoth MINLP

Page 32: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Branch-and-BoundOuter Approximation

Hybrid Methods

DefinitionBenders Decomposition

Outer Approximation (Duran and Grossmann, 1986)

Motivation: avoid solving huge number of NLPs• Exploit MILP/NLP solvers: decompose integer/nonlinear part

Key idea: reformulate MINLP as MILP (implicit)• Solve alternating sequence of MILP & NLP

NLP subproblem yj fixed:

NLP(yj)

minimize

xf(x, yj)

subject to c(x, yj) ≤ 0x ∈ X

Main Assumption: f , c are convex

(y )jNLP

MILP

Leyffer & Linderoth MINLP

Page 33: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Branch-and-BoundOuter Approximation

Hybrid Methods

DefinitionBenders Decomposition

Outer Approximation (Duran and Grossmann, 1986)

• let (xj , yj) solve NLP(yj)• linearize f , c about (xj , yj) =: zj• new objective variable η ≥ f(x, y)• MINLP (P ) ≡ MILP (M)

f(x)

η

(M)

minimizez=(x,y),η

η

subject to η ≥ fj +∇fTj (z − zj) ∀yj ∈ Y

0 ≥ cj +∇cTj (z − zj) ∀yj ∈ Yx ∈ X, y ∈ Y integer

SNAG: need all yj ∈ Y linearizations

Leyffer & Linderoth MINLP

Page 34: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Branch-and-BoundOuter Approximation

Hybrid Methods

DefinitionBenders Decomposition

Outer Approximation (Duran and Grossmann, 1986)

(Mk): lower bound (underestimate convex f , c)NLP(yj): upper bound U (fixed yj)

NLP( ) subproblemylinearizationNLP gives

MILP findsnew y

MILP infeasible?

Yes

STOP

No

MILP master program

⇒ stop, if lower bound ≥ upper bound

Leyffer & Linderoth MINLP

Page 35: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Branch-and-BoundOuter Approximation

Hybrid Methods

DefinitionBenders Decomposition

Outer Approximation & Benders Decomposition

Take OA cuts for zj := (xj , yj) ... wlog X = Rn

η ≥ fj +∇fTj (z − zj) & 0 ≥ cj +∇cTj (z − zj)

sum with (1, λj) ... λj multipliers of NLP(yj)

η ≥ fj + λTj cj + (∇fj +∇cjλj)

T (z − zj)

KKT conditions of NLP(yj) ⇒ ∇xfj +∇xcjλj = 0... eliminate x components from valid inequality in y

⇒ η ≥ fj + (∇yfj +∇ycjλj)T (y − yj)

NB: µj = ∇yfj +∇ycjλj multiplier of y = yj in NLP(yj)References: (Geoffrion, 1972)

Leyffer & Linderoth MINLP

Page 36: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Branch-and-BoundOuter Approximation

Hybrid Methods

LP/NLP Based Branch-and-BoundIntegrating SQP and Branch-and-Bound

LP/NLP Based Branch-and-Bound

AIM: avoid re-solving MILP master (M)

• Consider MILP branch-and-bound

• interrupt MILP, when yj found⇒ solve NLP(yj) get xj

• linearize f , c about (xj , yj)⇒ add linearization to tree

• continue MILP tree-search

... until lower bound ≥ upper bound

Leyffer & Linderoth MINLP

Page 37: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Branch-and-BoundOuter Approximation

Hybrid Methods

LP/NLP Based Branch-and-BoundIntegrating SQP and Branch-and-Bound

LP/NLP Based Branch-and-Bound

• need access to MILP solver ... call back exploit good MILP (branch-cut-price) solver (Akrotirianakis et al., 2001) use Gomory cuts in tree-search

• preliminary results: order of magnitude faster than OA same number of NLPs, but only one MILP

• similar ideas for Benders & Extended Cutting Plane methods

• recent implementation by CMU/IBM group

References: (Quesada and Grossmann, 1992)

Leyffer & Linderoth MINLP

Page 38: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Branch-and-BoundOuter Approximation

Hybrid Methods

LP/NLP Based Branch-and-BoundIntegrating SQP and Branch-and-Bound

Integrating SQP & Branch-and-Bound

AIM: Avoid solving NLP node to convergence.

Sequential Quadratic Programming (SQP)→ solve sequence (QPk) at every node

(QPk)

minimize

dfk +∇fT

k d+ 12d

THkd

subject to ck +∇cTk d ≤ 0xk + dx ∈ Xyk + dy ∈ Y .

Early branching:After QP step choose non-integral yk+1

i , branch & continue SQPReferences: (Borchers and Mitchell, 1994; Leyffer, 2001)

Leyffer & Linderoth MINLP

Page 39: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Branch-and-BoundOuter Approximation

Hybrid Methods

LP/NLP Based Branch-and-BoundIntegrating SQP and Branch-and-Bound

Integrating SQP & Branch-and-Bound

SNAG: (QPk) not lower bound⇒ no fathoming from upper bound

minimized

fk +∇fTk d+ 1

2dTHkd

subject to ck +∇cTk d ≤ 0xk + dx ∈ Xyk + dy ∈ Y .

Remedy: Exploit OA underestimating property (Leyffer, 2001):

• add objective cut fk +∇fTk d ≤ U − ε to (QPk)

• fathom node, if (QPk) inconsistent

NB: (QPk) inconsistent and trust-region active ⇒ do not fathom

Leyffer & Linderoth MINLP

Page 40: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Branch-and-BoundOuter Approximation

Hybrid Methods

LP/NLP Based Branch-and-BoundIntegrating SQP and Branch-and-Bound

Comparison of Classical MINLP Techniques

Summary of numerical experience

1. Quadratic OA master: usually fewer iterationMIQP harder to solve

2. NLP branch-and-bound faster than OA... depends on MIP solver

3. LP/NLP-based-BB order of magnitude faster than OA. . . also faster than B&B

4. Integrated SQP-B&B up to 3× faster than B&B' number of QPs per node

5. ECP works well, if function/gradient evals expensive

Leyffer & Linderoth MINLP

Page 41: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

Part III

Modern Developments in MINLP

Leyffer & Linderoth MINLP

Page 42: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

Importance of RelaxationsAggregation

Modern Methods for MINLP

1. Formulations• Relaxations• Good formulations: big M ′s and disaggregation

2. Cutting Planes• Cuts from relaxations and special structures• Cuts from integrality

3. Handling Nonconvexity• Envelopes• Methods

Leyffer & Linderoth MINLP

Page 43: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

Importance of RelaxationsAggregation

Relaxations

• z(S)def= minx∈S f(x)

• z(T )def= minx∈T f(x)

S

T

• Independent of f, S, T :z(T ) ≤ z(S)

• If x∗T = arg minx∈T f(x)

• And x∗T ∈ S, then

• x∗T = arg minx∈S f(x)

Leyffer & Linderoth MINLP

Page 44: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

Importance of RelaxationsAggregation

UFL: Uncapacitated Facility Location• Facilities: J

• Customers: I min∑j∈J

fjxj +∑i∈I

∑j∈J

fijyij

∑j∈J

yij = 1 ∀i ∈ I∑i∈I

yij ≤ |I|xj ∀j ∈ J (1)

OR yij ≤ xj ∀i ∈ I, j ∈ J (2)

• Which formulation is to be preferred?

• I = J = 40. Costs random.• Formulation 1. 53,121 seconds, optimal solution.• Formulation 2. 2 seconds, optimal solution.

Leyffer & Linderoth MINLP

Page 45: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

PreliminariesMILP Inequalities Applied to MINLPDisjunctive Inequalities

Valid Inequalities

• Sometimes we can get a better formulation by dynamicallyimproving it.

• An inequality πTx ≤ π0 is a valid inequality for S ifπTx ≤ π0 ∀x ∈ S

• Alternatively: maxx∈SπTx ≤ π0

• Thm: (Hahn-Banach). LetS ⊂ Rn be a closed, convex set,and let x 6∈ S. Then there existsπ ∈ Rn such that

πT x > maxx∈SπTx

S

xπTx = π0

Leyffer & Linderoth MINLP

Page 46: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

PreliminariesMILP Inequalities Applied to MINLPDisjunctive Inequalities

Nonlinear Branch-and-Cut

Consider MINLPminimize

x,yfT

x x+ fTy y

subject to c(x, y) ≤ 0y ∈ 0, 1p, 0 ≤ x ≤ U

• Note the Linear objective

• This is WLOG:

min f(x, y) ⇔ min η s.t. η ≥ f(x, y)

Leyffer & Linderoth MINLP

Page 47: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

PreliminariesMILP Inequalities Applied to MINLPDisjunctive Inequalities

It’s Actually Important!

• We want to approximate the convex hull of integer solutions,but without a linear objective function, the solution to therelaxation might occur in the interior.

• No Separating Hyperplane! :-(

min(y1 − 1/2)2 + (y2 − 1/2)2

s.t. y1 ∈ 0, 1, y2 ∈ 0, 1

η ≥ (y1 − 1/2)2 + (y2 − 1/2)2

y1

y2

(y1, y2)

η

Leyffer & Linderoth MINLP

Page 48: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

PreliminariesMILP Inequalities Applied to MINLPDisjunctive Inequalities

Valid Inequalities From Relaxations

• Idea: Inequalities valid for a relaxation are valid for original• Generating valid inequalities for a relaxation is often easier.

T

Sx

πTx

0

• Separation Problem over T:Given x, T find (π, π0) suchthat πT x > π0,πTx ≤ π0∀x ∈ T

Leyffer & Linderoth MINLP

Page 49: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

PreliminariesMILP Inequalities Applied to MINLPDisjunctive Inequalities

Simple Relaxations

• Idea: Consider one row relaxations

• If P = x ∈ 0, 1n | Ax ≤ b, then for any row i,Pi = x ∈ 0, 1n | aT

i x ≤ bi is a relaxation of P .

• If the intersection of the relaxations is a good approximationto the true problem, then the inequalities will be quite useful.

• Crowder et al. (1983) is the seminal paper that shows this tobe true for IP.

• MINLP: Single (linear) row relaxations are also valid ⇒ sameinequalities can also be used

Leyffer & Linderoth MINLP

Page 50: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

PreliminariesMILP Inequalities Applied to MINLPDisjunctive Inequalities

Knapsack Covers

K = x ∈ 0, 1n | aTx ≤ b

• A set C ⊆ N is a cover if∑

j∈C aj > b

• A cover C is a minimal cover if C \ j is not a cover ∀j ∈ C

• If C ⊆ N is a cover, then the cover inequality∑j∈C

xj ≤ |C| − 1

is a valid inequality for S

• Sometimes (minimal) cover inequalities are facets of conv(K)

Leyffer & Linderoth MINLP

Page 51: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

PreliminariesMILP Inequalities Applied to MINLPDisjunctive Inequalities

Other Substructures

• Single node flow: (Padberg et al., 1985)

S =

x ∈ R|N |+ , y ∈ 0, 1|N | |

∑j∈N

xj ≤ b, xj ≤ ujyj ∀ j ∈ N

• Knapsack with single continuous variable: (Marchand and

Wolsey, 1999)

S =

x ∈ R+, y ∈ 0, 1|N | |∑j∈N

ajyj ≤ b+ x

• Set Packing: (Borndorfer and Weismantel, 2000)

S =y ∈ 0, 1|N | | Ay ≤ e

A ∈ 0, 1|M |×|N |, e = (1, 1, . . . , 1)T

Leyffer & Linderoth MINLP

Page 52: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

PreliminariesMILP Inequalities Applied to MINLPDisjunctive Inequalities

The Chvatal-Gomory Procedure

• A general procedure for generating valid inequalities forinteger programs

• Let the columns of A ∈ Rm×n be denoted by a1, a2, . . . an• S = y ∈ Zn

+ | Ay ≤ b.1. Choose nonnegative multipliers u ∈ Rm

+

2. uTAy ≤ uT b is a valid inequality (∑

j∈N uTajyj ≤ uT b).

3.∑

j∈NbuTajcyj ≤ uT b (Since y ≥ 0).

4.∑

j∈NbuTajcyj ≤ buT bc is valid for S since buTajcyj is aninteger

• Simply Amazing: This simple procedure suffices to generateevery valid inequality for an integer program

Leyffer & Linderoth MINLP

Page 53: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

PreliminariesMILP Inequalities Applied to MINLPDisjunctive Inequalities

Extension to MINLP (Cezik and Iyengar, 2005)

• This simple idea also extends to mixed 0-1 conic programmingminimizez

def=(x,y)

fT z

subject to Az K by ∈ 0, 1p, 0 ≤ x ≤ U

• K: Homogeneous, self-dual, proper, convex cone

• x K y ⇔ (x− y) ∈ K

Leyffer & Linderoth MINLP

Page 54: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

PreliminariesMILP Inequalities Applied to MINLPDisjunctive Inequalities

Gomory On Cones (Cezik and Iyengar, 2005)

• LP: Kl = Rn+

• SOCP: Kq = (x0, x) | x0 ≥ ‖x‖• SDP: Ks = x = vec(X) | X = XT , X p.s.d

• Dual Cone: K∗ def= u | uT z ≥ 0 ∀z ∈ K

• Extension is clear from the following equivalence:

Az K b ⇔ uTAz ≥ uT b ∀u K∗ 0

• Many classes of nonlinear inequalities can be represented as

Ax Kq b or Ax Ks b

Leyffer & Linderoth MINLP

Page 55: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

PreliminariesMILP Inequalities Applied to MINLPDisjunctive Inequalities

Using Gomory Cuts in MINLP (Akrotirianakis et al., 2001)

• LP/NLP Based Branch-and-Bound solves MILP instances:minimizez

def=(x,y),η

η

subject to η ≥ fj +∇fTj (z − zj) ∀yj ∈ Y k

0 ≥ cj +∇cTj (z − zj) ∀yj ∈ Y k

x ∈ X, y ∈ Y integer

• Create Gomory mixed integer cuts from

η ≥ fj +∇fTj (z − zj)

0 ≥ cj +∇cTj (z − zj)

• Akrotirianakis et al. (2001) shows modest improvements

• Research Question: Other cut classes?

• Research Question: Exploit “outer approximation” property?

Leyffer & Linderoth MINLP

Page 56: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

PreliminariesMILP Inequalities Applied to MINLPDisjunctive Inequalities

Disjunctive Cuts for MINLP (Stubbs and Mehrotra, 1999)

Extension of Disjunctive Cuts for MILP: (Balas, 1979; Balas et al.,1993)

Continuous relaxation (zdef= (x, y))

• C def= z|c(z) ≤ 0, 0 ≤ y ≤ 1, 0 ≤ x ≤ U

• C def= conv(x ∈ C | y ∈ 0, 1p)

• C0/1j

def= z ∈ C|yj = 0/1

letMj(C)def=

z = λ0u0 + λ1u1

λ0 + λ1 = 1, λ0, λ1 ≥ 0u0 ∈ C0

j , u1 ∈ C1j

⇒ Pj(C) := projection ofMj(C) onto z

y

x

convexhull

⇒ Pj(C) = conv (C ∩ yj ∈ 0, 1) and P1...p(C) = CLeyffer & Linderoth MINLP

Page 57: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

PreliminariesMILP Inequalities Applied to MINLPDisjunctive Inequalities

Disjunctive Cuts: Example

minimizex,y

x | (x− 1/2)2 + (y − 3/4)2 ≤ 1,−2 ≤ x ≤ 2, y ∈ 0, 1

C0j C1

j

z = (x, y)

y

xGiven z with yj 6∈ 0, 1 find separatinghyperplane

minimize

z‖z − z‖

subject to z ∈ Pj(C)

Leyffer & Linderoth MINLP

Page 58: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

PreliminariesMILP Inequalities Applied to MINLPDisjunctive Inequalities

Disjunctive Cuts ExampleC0

j C1j

z = (x, y)

z∗

z∗def= arg min ‖z − z‖

s.t. λ0u0 + λ1u1 = zλ0 + λ1 = 1(

−0.160

)≤ u0 ≤

(0.661

)(−0.47

0

)≤ u1 ≤

(1.471

)λ0, λ1 ≥ 0

NONCONVEX

Leyffer & Linderoth MINLP

Page 59: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

PreliminariesMILP Inequalities Applied to MINLPDisjunctive Inequalities

What to do? (Stubbs and Mehrotra, 1999)

• Look at the perspective of c(z)

P(c(z), µ) = µc(z/µ)

• Think of z = µz

• Perspective gives a convex reformulation ofMj(C): Mj(C),where

C :=

(z, µ)

∣∣∣∣∣∣µci(z/µ) ≤ 00 ≤ µ ≤ 10 ≤ x ≤ µU, 0 ≤ y ≤ µ

• c(0/0) = 0 ⇒ convex representation

Leyffer & Linderoth MINLP

Page 60: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

PreliminariesMILP Inequalities Applied to MINLPDisjunctive Inequalities

Disjunctive Cuts Example

C =

x

∣∣∣∣∣∣∣∣µ

[(x/µ− 1/2)2 + (y/µ− 3/4)2 − 1

]≤ 0

−2µ ≤ x ≤ 2µ0 ≤ y ≤ µ0 ≤ µ ≤ 1

C0j C1

j y

x

µ

C0j

C1jy

x

µ

C0j

C1j

y

x

µ

Leyffer & Linderoth MINLP

Page 61: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

PreliminariesMILP Inequalities Applied to MINLPDisjunctive Inequalities

Example, cont.

C0j = (z, µ) | yj = 0 C1

j = (z, µ) | yj = µ

• Take v0 ← µ0u0 v1 ← µ1u1

min ‖z − z‖

s.t. v0 + v1 = zµ0 + µ1 = 1(v0, µ0) ∈ C0

j

(v1, µ1) ∈ C1j

µ0, µ1 ≥ 0

Solution to example:(x∗

y∗

)=

(−0.4010.780

)

• separating hyperplane: ψT (z − z), where ψ ∈ ∂‖z − z‖

Leyffer & Linderoth MINLP

Page 62: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

PreliminariesMILP Inequalities Applied to MINLPDisjunctive Inequalities

Example, Cont.C0j C1

j

z = (x, y)

z∗

0.198x+ 0.061y = −0.032

ψ =

(2x∗ + 0.52y∗ − 0.75

)0.198x+ 0.061y ≥ −0.032

Leyffer & Linderoth MINLP

Page 63: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

PreliminariesMILP Inequalities Applied to MINLPDisjunctive Inequalities

Nonlinear Branch-and-Cut (Stubbs and Mehrotra, 1999)

• Can do this at all nodes of the branch-and-bound tree

• Generalize disjunctive approach from MILP• solve one convex NLP per cut

• Generalizes Sherali and Adams (1990) and Lovasz andSchrijver (1991)

• tighten cuts by adding semi-definite constraint

• Stubbs and Mehrohtra (2002) also show how to generateconvex quadratic inequalities, but computational results arenot that promising

Leyffer & Linderoth MINLP

Page 64: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

PreliminariesMILP Inequalities Applied to MINLPDisjunctive Inequalities

Generalized Disjunctive Programming (Raman and

Grossmann, 1994; Lee and Grossmann, 2000)

Consider disjunctive NLP

minimizex,Y

∑fi + f(x)

subject to

Yi

ci(x) ≤ 0fi = γi

∨ ¬Yi

Bix = 0fi = 0

∀i ∈ I0 ≤ x ≤ U, Ω(Y ) = true, Y ∈ true, falsep

convex hull representation ...

x = vi1 + vi0, λi1 + λi0 = 1λi1ci(vi1/λi1) ≤ 0, Bivi0 = 00 ≤ vij ≤ λijU, 0 ≤ λij ≤ 1, fi = λi1γi

Leyffer & Linderoth MINLP

Page 65: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

DifficultiesEnvelopesBilinear Terms

Dealing with Nonconvexities

• Functional nonconvexitycauses serious problems.

• Branch and bound musthave true lower bound(global solution)

• Underestimate nonconvexfunctions. Solve relaxation.Provides lower bound.

• If relaxation is not exact,then branch

Leyffer & Linderoth MINLP

Page 66: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

DifficultiesEnvelopesBilinear Terms

Dealing with Nonconvex Constraints

• If nonconvexity inconstraints, may need tooverestimate andunderestimate the functionto get a convex region

Leyffer & Linderoth MINLP

Page 67: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

DifficultiesEnvelopesBilinear Terms

Envelopes

f : Ω→ R

• Convex Envelope (vexΩ(f)):Pointwise supremum ofconvex underestimators of fover Ω.

• Concave Envelope(cavΩ(f)): Pointwiseinfimum of concaveoverestimators of f over Ω.

Leyffer & Linderoth MINLP

Page 68: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

DifficultiesEnvelopesBilinear Terms

Branch-and-Bound Global Optimization Methods

• Under/Overestimate “simple” parts of (Factorable) Functionsindividually

• Bilinear Terms• Trilinear Terms• Fractional Terms• Univariate convex/concave terms

• General nonconvex functions f(x) can be underestimated overa region [l, u] “overpowering” the function with a quadraticfunction that is ≤ 0 on the region of interest

L(x) = f(x) +n∑

i=1

αi(li − xi)(ui − xi)

Refs: (McCormick, 1976; Adjiman et al., 1998; Tawarmalani andSahinidis, 2002)

Leyffer & Linderoth MINLP

Page 69: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

DifficultiesEnvelopesBilinear Terms

Bilinear Terms

The convex and concave envelopes of the bilinear function xy overa rectangular region

Rdef= (x, y) ∈ R2 | lx ≤ x ≤ ux, ly ≤ y ≤ uy

are given by the expressions

vexxyR(x, y) = maxlyx+ lxy − lxly, uyx+ uxy − uxuycavxyR(x, y) = minuyx+ lxy − lxuy, lyx+ uxy − uxly

Leyffer & Linderoth MINLP

Page 70: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

DifficultiesEnvelopesBilinear Terms

Worth 1000 Words?

Leyffer & Linderoth MINLP

Page 71: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

FormulationsInequalities

Dealing with Nonconvexity

DifficultiesEnvelopesBilinear Terms

Summary

• MINLP: Good relaxations are important

• Relaxations can be improved• Statically: Better formulation/preprocessing• Dynamically: Cutting planes

• Nonconvex MINLP:• Methods exist, again based on relaxations

• Tight relaxations is an active area of research

• Lots of empirical questions remain

Leyffer & Linderoth MINLP

Page 72: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Special Ordered SetsImplementation & Software Issues

Part IV

Implementation and Software

Leyffer & Linderoth MINLP

Page 73: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Special Ordered SetsImplementation & Software Issues

Implementation and Software for MINLP

1. Special Ordered Sets

2. Implementation & Software Issues

Leyffer & Linderoth MINLP

Page 74: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Special Ordered SetsImplementation & Software Issues

Special Ordered Sets of Type 1Special Ordered Sets of Type 2Special Ordered Sets of Type 3

Special Ordered Sets of Type 1

SOS1:∑λi = 1 & at most one λi is nonzero

Example 1: d ∈ d1, . . . , dp discrete diameters

⇔ d =∑λidi and λ1, . . . , λp is SOS1

⇔ d =∑λidi and

∑λi = 1 and λi ∈ 0, 1

. . . d is convex combination with coefficients λi

Example 2: nonlinear function c(y) of single integer

⇔ y =∑iλi and c =

∑c(i)λi and λ1, . . . , λp is SOS1

References: (Beale, 1979; Nemhauser, G.L. and Wolsey, L.A.,1988; Williams, 1993) . . .

Leyffer & Linderoth MINLP

Page 75: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Special Ordered SetsImplementation & Software Issues

Special Ordered Sets of Type 1Special Ordered Sets of Type 2Special Ordered Sets of Type 3

Special Ordered Sets of Type 1

SOS1:∑λi = 1 & at most one λi is nonzero

Branching on SOS1

1. reference row a1 < . . . < ap

e.g. diameters

2. fractionality: a :=∑aiλi

3. find t : at < a ≤ at+1

4. branch: λt+1, . . . , λp = 0or λ1, . . . , λt = 0 a < a

ta > a

t+1

Leyffer & Linderoth MINLP

Page 76: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Special Ordered SetsImplementation & Software Issues

Special Ordered Sets of Type 1Special Ordered Sets of Type 2Special Ordered Sets of Type 3

Special Ordered Sets of Type 2

SOS2:∑λi = 1 & at most two adjacent λi nonzero

Example: Approximation of nonlinear function z = z(x)

z(x)

x

• breakpoints x1 < . . . < xp

• function values zi = z(xi)

• piece-wise linear

• x =∑λixi

• z =∑λizi

• λ1, . . . , λp is SOS2

. . . convex combination of two breakpoints . . .

Leyffer & Linderoth MINLP

Page 77: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Special Ordered SetsImplementation & Software Issues

Special Ordered Sets of Type 1Special Ordered Sets of Type 2Special Ordered Sets of Type 3

Special Ordered Sets of Type 2

SOS2:∑λi = 1 & at most two adjacent λi nonzero

Branching on SOS2

1. reference row a1 < . . . < ap

e.g. ai = xi

2. fractionality: a :=∑aiλi

3. find t : at < a ≤ at+1

4. branch: λt+1, . . . , λp = 0or λ1, . . . , λt−1 t

x > ax < at

Leyffer & Linderoth MINLP

Page 78: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Special Ordered SetsImplementation & Software Issues

Special Ordered Sets of Type 1Special Ordered Sets of Type 2Special Ordered Sets of Type 3

Special Ordered Sets of Type 3

Example: Approximation of 2D function u = g(v, w)

Triangularization of [vL, vU ]× [wL, wU ] domain

1. vL = v1 < . . . < vk = vU

2. wL = w1 < . . . < wl = wU

3. function uij := g(vi, wj)

4. λij weight of vertex (i, j)

• v =∑λijvi

• w =∑λijwj

• u =∑λijuij

v

w

1 =∑λij is SOS3 . . .

Leyffer & Linderoth MINLP

Page 79: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Special Ordered SetsImplementation & Software Issues

Special Ordered Sets of Type 1Special Ordered Sets of Type 2Special Ordered Sets of Type 3

Special Ordered Sets of Type 3

SOS3:∑λij = 1 & set condition holds

1. v =∑λijvi ... convex combinations

2. w =∑λijwj

3. u =∑λijuij

λ11, . . . , λkl satisfies set condition

⇔ ∃ trangle ∆ : (i, j) : λij > 0 ⊂ ∆ v

w

violates set condn

i.e. nonzeros in single triangle ∆

Leyffer & Linderoth MINLP

Page 80: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Special Ordered SetsImplementation & Software Issues

Special Ordered Sets of Type 1Special Ordered Sets of Type 2Special Ordered Sets of Type 3

Branching on SOS3

λ violates set condition

• compute centers:v =

∑λijvi &

w =∑λijwi

• find s, t such thatvs ≤ v < vs+1 &ws ≤ w < ws+1

• branch on v or w

v

w

= center of gravity

B

T

vertical branching:∑L

λij = 1∑R

λij = 1 horizontal

branching:∑T

λij = 1∑B

λij = 1

Leyffer & Linderoth MINLP

Page 81: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Special Ordered SetsImplementation & Software Issues

Special Ordered Sets of Type 1Special Ordered Sets of Type 2Special Ordered Sets of Type 3

Extension to SOS-k

Example: electricity transmission network:

c(x) = 4x1 − x22 − 0.2 · x2x4 sin(x3)

(Martin et al., 2005) extend SOS3 to SOSk models for any k⇒ function with p variables on N grid needs Np λ’s

Alternative (Gatzke, 2005):

• exploit computational graph' automatic differentiation

• only need SOS2 & SOS3 ...replace nonconvex parts

• piece-wise polyhedral approx.

Leyffer & Linderoth MINLP

Page 82: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Special Ordered SetsImplementation & Software Issues

MINLP Software

Software for MINLP

• Outer Approximation: DICOPT++ (& AIMMS)NLP solvers: CONOPT, MINOS, SNOPTMILP solvers: CPLEX, OSL2

• Branch-and-Bound Solvers: SBB & MINLPNLP solvers: CONOPT, MINOS, SNOPT & FilterSQPvariable & node selection; SOS1 & SOS2 support

• Global MINLP: BARON & MINOPT underestimators &branchingCPLEX, MINOS, SNOPT, OSL

• Online Tools: MINLP World, MacMINLP & NEOS MINLPWorld www.gamsworld.org/minlp/NEOS server www-neos.mcs.anl.gov/

Leyffer & Linderoth MINLP

Page 83: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Special Ordered SetsImplementation & Software Issues

MINLP Software

COIN-OR

http://www.coin-or.org

• COmputational INfrastructure for Operations Research

• A library of (interoperable) software tools for optimization

• A development platform for open source projects in the ORcommunity

• Possibly Relevant Modules:• OSI: Open Solver Interface• CGL: Cut Generation Library• CLP: Coin Linear Programming Toolkit• CBC: Coin Branch and Cut• IPOPT: Interior Point OPTimizer for NLP• NLPAPI: NonLinear Programming API

Leyffer & Linderoth MINLP

Page 84: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Special Ordered SetsImplementation & Software Issues

MINLP Software

MINLP with COIN-OR

New implementation of LP/NLP based BB

• MIP branch-and-cut: CBC & CGL

• NLPs: IPOPT interior point ... OK for NLP(yi)

• New hybrid method:• solve more NLPs at non-integer yi

⇒ better outer approximation• allow complete MIP at some nodes⇒ generate new integer assignment

... faster than DICOPT++, SBB

• simplifies to OA and BB at extremes ... less efficient

... see Bonami et al. (2005) ... coming in 2006.

Leyffer & Linderoth MINLP

Page 85: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Special Ordered SetsImplementation & Software Issues

MINLP Software

Conclusions

MINLP rich modeling paradigm most popular solver on NEOS

Algorithms for MINLP: Branch-and-bound (branch-and-cut) Outer approximation et al.

“MINLP solvers lag 15 years behind MIP solvers”

⇒ many research opportunities!!!

Leyffer & Linderoth MINLP

Page 86: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

Part V

References

Leyffer & Linderoth MINLP

Page 87: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

References

C. Adjiman, S. Dallwig, C. A. Floudas, and A. Neumaier. A global optimizationmethod, aBB, for general twice-differentiable constrained NLPs - I. Theoreticaladvances. Computers and Chemical Engineering, 22:1137–1158, 1998.

I. Akrotirianakis, I. Maros, and B. Rustem. An outer approximation basedbranch-and-cut algorithm for convex 0-1 MINLP problems. Optimization Methodsand Software, 16:21–47, 2001.

E. Balas. Disjunctive programming. In Annals of Discrete Mathematics 5: DiscreteOptimization, pages 3–51. North Holland, 1979.

E. Balas, S. Ceria, and G. Corneujols. A lift-and-project cutting plane algorithm formixed 0-1 programs. Mathematical Programming, 58:295–324, 1993.

E. M. L. Beale. Branch-and-bound methods for mathematical programming systems.Annals of Discrete Mathematics, 5:201–219, 1979.

P. Bonami, L. Biegler, A. Conn, G. Cornuejols, I. Grossmann, C. Laird, J. Lee,A. Lodi, F. Margot, N. Saaya, and A. Wachter. An algorithmic framework forconvex mixed integer nonlinear programs. Technical report, IBM Research Division,Thomas J. Watson Research Center, 2005.

B. Borchers and J. E. Mitchell. An improved branch and bound algorithm for MixedInteger Nonlinear Programming. Computers and Operations Research, 21(4):359–367, 1994.

R. Borndorfer and R. Weismantel. Set packing relaxations of some integer programs.Mathematical Programming, 88:425 – 450, 2000.

M. T. Cezik and G. Iyengar. Cuts for mixed 0-1 conic programming. MathematicalProgramming, 2005. to appear.

Leyffer & Linderoth MINLP

Page 88: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

References

H. Crowder, E. L. Johnson, and M. W. Padberg. Solving large scale zero-one linearprogramming problems. Operations Research, 31:803–834, 1983.

D. De Wolf and Y. Smeers. The gas transmission problem solved by an extension ofthe simplex algorithm. Management Science, 46:1454–1465, 2000.

M. Duran and I. E. Grossmann. An outer-approximation algorithm for a class ofmixed–integer nonlinear programs. Mathematical Programming, 36:307–339,1986.

A. M. Geoffrion. Generalized Benders decomposition. Journal of Optimization Theoryand Applications, 10:237–260, 1972.

I. E. Grossmann and R. W. H. Sargent. Optimal design of multipurpose batch plants.Ind. Engng. Chem. Process Des. Dev., 18:343–348, 1979.

Harjunkoski, I., Westerlund, T., Porn, R. and Skrifvars, H. Different transformationsfor solving non-convex trim-loss problems by MINLP. European Journal ofOpertational Research, 105:594–603, 1998.

Jain, V. and Grossmann, I.E. Cyclic scheduling of continuous parallel-process unitswith decaying performance. AIChE Journal, 44:1623–1636, 1998.

G. R. Kocis and I. E. Grossmann. Global optimization of nonconvex mixed–integernonlinear programming (MINLP) problems in process synthesis. IndustrialEngineering Chemistry Research, 27:1407–1421, 1988.

S. Lee and I. Grossmann. New algorithms for nonlinear disjunctive programming.Computers and Chemical Engineering, 24:2125–2141, 2000.

S. Leyffer. Integrating SQP and branch-and-bound for mixed integer nonlinearprogramming. Computational Optimization & Applications, 18:295–309, 2001.

Leyffer & Linderoth MINLP

Page 89: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

References

L. Lovasz and A. Schrijver. Cones of matrices and setfunctions, and 0-1 optimization.SIAM Journal on Optimization, 1, 1991.

H. Marchand and L. Wolsey. The 0-1 knapsack problem with a single continuousvariable. Mathematical Programming, 85:15–33, 1999.

A. Martin, M. Moller, and S. Moritz. Mixed integer models for the stationary case ofgas network optimization. Technical report, Darmstadt University of Technology,2005.

G. P. McCormick. Computability of global solutions to factorable nonconvexprograms: Part I—Convex underestimating problems. MathematicalProgramming, 10:147–175, 1976.

Nemhauser, G.L. and Wolsey, L.A. Integer and Combinatorial Optimization. JohnWiley, New York, 1988.

M. Padberg, T. J. Van Roy, and L. Wolsey. Valid linear inequalities for fixed chargeproblems. Operations Research, 33:842–861, 1985.

I. Quesada and I. E. Grossmann. An LP/NLP based branch–and–bound algorithm forconvex MINLP optimization problems. Computers and Chemical Engineering, 16:937–947, 1992.

Quist, A.J. Application of Mathematical Optimization Techniques to NuclearReactor Reload Pattern Design. PhD thesis, Technische Universiteit Delft,Thomas Stieltjes Institute for Mathematics, The Netherlands, 2000.

R. Raman and I. E. Grossmann. Modeling and computational techniques for logicbased integer programming. Computers and Chemical Engineering, 18:563–578,1994.

Leyffer & Linderoth MINLP

Page 90: Mixed Integer Nonlinear Programming - Lehigh Universitycoral.ise.lehigh.edu/wp-content/uploads/presentations/informs-05-minlp-handout.pdfmixed integer nonlinear programming problem

References

H. D. Sherali and W. P. Adams. A hierarchy of relaxations between the continuousand convex hull representations for zero-one programming problems. SIAM Journalon Discrete Mathematics, 3:411–430, 1990.

O. Sigmund. A 99 line topology optimization code written in matlab. StructuralMultidisciplinary Optimization, 21:120–127, 2001.

R. Stubbs and S. Mehrohtra. Generating convex polynomial inequalities for mixed 0-1programs. Journal of Global Optimization, 24:311–332, 2002.

R. A. Stubbs and S. Mehrotra. A branch–and–cut method for 0–1 mixed convexprogramming. Mathematical Programming, 86:515–532, 1999.

M. Tawarmalani and N. V. Sahinidis. Convexification and Global Optimization inContinuous and Mixed-Integer Nonlinear Programming: Theory, Algorithms,Software, and Applications. Kluwer Academic Publishers, Boston MA, 2002.

J. Viswanathan and I. E. Grossmann. Optimal feed location and number of trays fordistillation columns with multiple feeds. I&EC Research, 32:2942–2949, 1993.

Westerlund, T., Isaksson, J. and Harjunkoski, I. Solving a production optimizationproblem in the paper industry. Report 95–146–A, Department of ChemicalEngineering, Abo Akademi, Abo, Finland, 1995.

Westerlund, T., Pettersson, F. and Grossmann, I.E. Optimization of pumpconfigurations as MINLP problem. Computers & Chemical Engineering, 18(9):845–858, 1994.

H. P. Williams. Model Solving in Mathematical Programming. John Wiley & SonsLtd., Chichester, 1993.

Leyffer & Linderoth MINLP