IOE 610/Math 660: Linear Programming IImepelman/teaching/IOE610/Handouts/610In… · IOE 610/Math...

IOE 610/Math 660: Linear Programming II

Marina A. Epelman

Fall 2013

IOE 610: LP II, Fall 2013

Linear Programming II

I Lectures: Tuesdays and Thursdays, 12:00–1:30I Instructor: Marina A. Epelman

”[email protected]”, o�ce: 2845 IOE; O�ce hours: asannounced on CTools, and by appointment

I Textbook: Introduction to Linear Optimization by Bertsimasand Tsitsiklis and AMPL by Fourer, Gay and Kernighan;additional readings will be posted online

I Other course materials, including assignments, lecture topicsand these slides are available on CTools. Unrestrictedresources (excluding assignments, solutions and copyrightedmaterials) are at

www-personal.umich.edu/⇠mepelman/teaching/IOE610/I Required background: Working knowledge of linear algebra

and analysis and a general level of mathematical maturity.Exposure to numerical computing, optimization, and itsapplication fields is helpful but not required.

IOE 610: LP II, Fall 2013 Course Introduction Page 2

Course Logistics

The following have equal weights in determining the grade:

I Weekly homeworks

I A takehome exam approximately at midpoint of the semester

I A takehome exam during the finals week

All of the above will to some extent involve computational work inAMPL and Matlab.Individual work policy: You are allowed (indeed, encouraged) todiscuss homework problems with other students and read booksand papers to learn more about the subject. However, the worksubmitted by you should be your own writing, and if programmingis involved, you will be responsible for the correctness of the codeused and reported computational results. Copying or rephrasing ofanother person’s answers is unacceptable. Exam policies will beannounced separately.Please read syllabus for complete course policies.


Informal (and tentative) course outline

I Mathematical modeling and Linear ProgrammingI Geometry and theory of LP problems

I polyhedral geometryI properties of optimal solutionsI LP duality

I Algorithms for solving LPs and their complexity:I Simplex method and its variantsI Ellipsoid algorithmI Interior point methodsI Decomposition methods for large problems with structure

I Special cases and extensions:I Network flow problemsI Integer linear programs


Optimization problems

An optimization problem is a

I Set of decisions to make

I A single quantitative criterion to compare decisions

I Rules governing interactions of decisions

Example: the Diet ProblemGiven a collection of foods (e.g., all items in your favoritesupermarket),...

I Decide how much of each item to eat in a given day,

I ... with the objective of choosing the cheapest daily diet,...

I ... while meeting all the rules/constraints governingnutritional requirements (not too much fat, not too manycalories, su�cient amount of protein, etc.)

IOE 610: LP II, Fall 2013 Optimization problems Page 5

Diet problem: some history

I Posed by Stigler in 1945I Solved heuristically

I Dantzig 1963, 1990I Solved optimally in 1947 using simplex methodI 120 man-hours

I Garille and Gass 2001I Solved using current dietary guidelines and pricesI Solution (“optimal diet”): 1.31 cups wheat flour, 1.32 cups

rolled oats, 16 oz. milk, 3.86 tbsp peanut butter, 7.28 tbsplard, 0.0108 oz. beef, 1.77 bananas, 0.0824 oranges, 0.707cups cabbage, 0.314 carrots, 0.387 potatoes, 0.53 cups porkand beans


Diet problem: lessons learned

I Decision making in a complex system:I Very large number of choices to considerI Many rules to satisfyI Decisions interact

I The diet problem is a concrete illustration of a wide array ofideas and concepts:

I Need good sources of data to pose the problem (how muchprotein in 1 cup of wheat flour?)

I Distinguish optimization problem from problem instance:I

Diet problem: the conceptual problem of choosing how muchof available items to eat in order to minimize cost whilesatisfying nutritional constraints.

I A specific instance of the diet problem: depends on whatitems are sold at your store, how much they cost, and whatyour specific nutritional needs are.

I Allows for mathematical insight:I Isn’t it surprising that out of all food items available, only 12

are in the optimal diet?


Mathematical Program

A Mathematical Program1 or, in more modern terminology, aMathematical Optimization Problem, is a mathematicalrepresentation, or model, of an optimization problem in whichdecisions that need to be made are quantitative.

I Decisions , Decision variables

I Comparison criterion , Objective function

I Rules governing interactions , Constraints

1Note the outdated use of the term “program” — it means “plan,” or“schedule,” not “computer program” — the term was coined in the 1940s.

IOE 610: LP II, Fall 2013 Mathematical Programming Page 8

Mathematical Programming model of the Diet ProblemA simple version of the diet problem:

I A student is concerned about his nutritionI He decides to eat a combination of various cereals as a snack,

to get at least the minimum recommended amount of themain nutrients

I His objective is to minimize the cost of this snackI Need the following data:

I The list of cereals available (or palatable!) — n totalI The list of nutrients — m totalI b

i

— minimum amount of nutrient i , i = 1, . . . ,mI c

j

— cost of one serving of cereal j , j = 1, . . . , nI a

ij

— amount of vitamin i in one serving of cereal j , 8i , jI “8” = “for all”

I The student’s problem:

(Student’s problem) mininimizeP

n

j=1 cjxjs.t.

Pn

j=1 aijxj � bi

8ixj

� 0 8jAMPL file: cerealdiet.modIOE 610: LP II, Fall 2013 A simple diet problem Page 9

Instance of the diet problemStudent’s diet problem data

NutrientMinimum*daily*amount

Total&Raisin&Bran

Whole&Grain&Total

Multi3Grain&Cheerios

Kashi&Good&friends

Kashi&Heart&to&Heart,&wild&blueberrry

Fiber 25*gram 5 3 2.8 11.8 4Vitamin&C 60*mg 0 60 15.5 0 30Iron 18*mg 18 18 18.6 1.6 1.8Calcium 1000*mg 1000 1000 104 18 24.7Vitamin&E 20*mg 13.5 13.5 14 0.4 20Vitamin&B6 2*mcg 2 2 2.1 0.1 2Vitamin&12 6*mcg 6 6 6.2 0 6Price $*per*serving 0.39$&&&&&&& 0.30$&&&&&&&& 0.35$&&&&&&&&& 0.52$&&&&&&&&& 0.45$&&&&&&&&&&&&

Nutrient*amount*per*serving

AMPL files cerealdiet.mod, cerealnutr.dat andcerealdiet.run

IOE 610: LP II, Fall 2013 A simple diet problem Page 10

Related: Pharmacist’s pricing problem

I A pharmacist wants to convince this student, and others likehim, to stop munching on dry cereal and start takingsupplement pills instead.

I Pharmacist needs to decide how much to charge for her pillsI The pills need to be cheap to compete with the cereals,I ... but she wants to maximize her profits!

I Pharmacist’s problem formulation:I Variables: p

i

� 0 is the selling price of one unit ofvitamin/nutrient i in pill form

I Constraints: To be competitive with cereal j ,Pm

i=1 aijpi cj

8jI Objective: Pharmacist’s revenue:

Pm

i=1 bipiI So, the pharmacist’s problem is

(Pharmacist’s problem) maximizeP

m

i=1 pibis.t.

Pm

i=1 aijpi cj

8jpi

� 0 8i

IOE 610: LP II, Fall 2013 A simple diet problem Page 11

Linear Programs (LPs)

I Notice that models of both the Diet Problem and the PricingProblem had the following properties:

I Continuous variablesI Linear objective functionI Constraints are linear inequalities

I Such mathematical programs are called Linear Programmingproblems, or Linear Optimization problems, or LinearPrograms, or LPs, and they are the subject of this course.

I Despite seeming simplicity, widely applicable in the real world

I Have beautiful mathematical properties, which allowed thediscovery of fast algorithms for solving even very largeinstances

I Provide the foundation for analysis and solution methods formore general mathematical programs (i.e., non-linear, or oneswith discrete variables)

IOE 610: LP II, Fall 2013 Linear Programming Page 12

Areas of applications of LP models and methods

E�cient resource allocation, military operations planning,production and inventory planning, capacity expansion,manufacturing process design, sta↵ scheduling, location planning,tra�c routing, supply chain management, economic game theory,airline crew and plane scheduling, telecommunication capacityallocation and network design, medical treatment planning, imagereconstruction, publishing (typesetting), finance (asset allocation),mathematics (as a proof technique and computational method),data analysis, pattern classification, optimal control, mechanicalstructure design, electromagnetic antennae design, etc.

Read sections 1.2-1.3 for examples!


What is a linear programming problem?

minimizex1,x2,x3,x4 2x1 �x2 +4x3

subject to x1 +x2 +x4 23x2 �x3 = 5

x3 +x4 � 3x1 � 0

x3 0

A linear programming problem (or a linear program, or an LP)is the problem of minimizing a linear cost function subject to linearequality and inequality constraints on the variables.


A general LP

Given a vector of “costs” c 2 <n; finite index sets M1, M2, M3

and, for each i 2 M1 [M2 [M3, a vector ai

2 <n and a scalar bi

;subsets N1 and N2 of {1, . . . , n} indicating which variables arerestricted in sign

minimizex (or maximizex) cTxs.t. aT

i

x � bi

, i 2 M1

aTi

x bi

, i 2 M2

aTi

x = bi

, i 2 M3

xj

� 0, j 2 N1

xj

0, j 2 N2


A few terms of the trade

I x1, . . . , xn — decision variables

I If j 62 N1 [ N2, xj is unrestricted, or free, variable

I Vector x = (x1, . . . , xn)T satisfying all the constraints —feasible solution, or feasible vector, or feasible point

I Set of all feasible solutions — feasible region, or feasible set

I If no feasible solutions exist, we are facing an infeasibleproblem

I The function cTx — objective, or cost, function

I A feasible solution x? that minimizes the objective functionover the feasible region — optimal solution

I The value cTx? — optimal cost

I If for every K 2 < there exists a feasible x such thatcTx < K , the optimal cost is �1, or unbounded below (orthe problem is unbounded).


Combinatorial auction description2

I There are m mutually exclusive potential states at maturitytime. Exactly one state will be true at maturity.

I An auctioneer is prepared to issue contracts on bids asfollows:

I Each bid should specify a subset of statesI The bidder will receive $w if the final state is one of those

specified in the bid, and $0 o/w

I Each auction participant j (out of n) places orders with theauctioneer, with the order consisting of:

I Bid vector Aj

= (a1j , a2j , . . . , amj

)T , with each componenteither 0 or 1

I Number $⇡j

, which is the price participant j is willing to payfor 1 unit of this order

I Number qj

, which is the maximum number of units of thisorder he/she is willing to buy

2From Linear and Nonlinear Programming by Luenberger and YeIOE 610: LP II, Fall 2013 Examples of LP models: Combinatorial Auction Page 17

Combinatorial auctionOrder fills

I Upon receiving all n orders, auctioneer decidesxj

, j = 1, . . . , n — number of units awarded to jth orderI Must have 0 x

j

qj

I jth bidder will pay ⇡j

xj

for the contractI If i is the winning state, jth bidder will receive w · (a

ij

xj

) fromthe auctioneer

I Net profit for the auctioneer (assuming i is the winning state):

nX

j=1

⇡j

xj

� w

0

@nX

j=1

aij

xj

1

A = ⇡Tx� w(aTi

x),

where ai

= (ai1, ai2, . . . , ain)T

IOE 610: LP II, Fall 2013 Examples of LP models: Combinatorial Auction Page 18

Combinatorial auctionWorst-case profit maximization

I A risk-averse auctioneer would want to maximize worst-caseprofit:

maximizex ⇡Tx� w ·maxi

{aTi

x}s.t. 0 x q

I Does not look like an LP!

IOE 610: LP II, Fall 2013 Examples of LP models: Combinatorial Auction Page 19

Convex functions

Definition 1.1

(a) A function f : <n ! < is called convex if 8x, y 2 <n, and8� 2 [0, 1],

f (�x+ (1� �)y) �f (x) + (1� �)f (y)

(b) A function f : <n ! < is called concave if 8x, y 2 <n, and8� 2 [0, 1],

f (�x+ (1� �)y) � �f (x) + (1� �)f (y)

Theorem 1.1

Let f1, . . . , fm : <n ! < be convex functions. Then, the function fdefined by f (x) = max

i=1,...,m fi

(x) is convex.

IOE 610: LP II, Fall 2013 Modeling with PWLC functions Page 20

Linear, a�ne and piecewise linear convex functions

Linear function

f (x) = c0x, where c 2 <n, is a linear function of x.

I By formal definition, a function f : <n ! < is linear ifI for any x 2 <n and any ↵ 2 <, f (↵x) = ↵f (x), andI for any x, y 2 <n, f (x+ y) = f (x) + f (y)

I It can be proven that for any linear function f : <n ! < wecan find c 2 <n such that f (x) = c0x.

A�ne function

f (x) = c0x+ d , where c 2 <n and d 2 <, is an a�ne function of x.

PWLC function

Let c1, . . . , cm 2 <n and d1, . . . , dm 2 <.f (x) = max

i=1,...,m(c0i

x+ di

) is a piecewise linear convex function.


Using LP to solve problems with PWLC cost functions

minimizex maxi=1,...,m(c0

i

x+ di

)s.t. linear constraints on x

is equivalent to:

minimizex,z zs.t. z � c0

i

x+ di

, i = 1, . . . ,mlinear constraints on x

Two mathematical programs, (P1) and (P2) are equivalent if

I They are either both infeasible, or have the same optimalcost, and

I Given an optimal solution of (P1), we can (easily) calculatean optimal solution of (P2), and vice versa


Back to the combinatorial auctionWorst-case profit maximization

I Recall: A risk-averse auctioneer maximizes worst-case profit:

maximizex ⇡Tx� w ·maxi

{aTi

x}s.t. 0 x q

I Equivalent LP:

maximizex,s ⇡Tx� w · ss.t. aT

i

x� s 0, i = 1, . . . ,m0 x q,

or:maximizex,s ⇡Tx� w · s

s.t. Ax� e · s 00 x q,

whereI where A 2 <m⇥n is a matrix with columns A

j

(or rows aTi

)I e 2 <m is a vector of all ones

I Note that x = 0 is feasible, so the profit is always � 0IOE 610: LP II, Fall 2013 Modeling with PWLC functions Page 23

Background on radiation therapy in cancer treatment

I This year, 1,200,000 Americans will be diagnosed with cancerI 600,000+ patients will receive radiation therapy

I beam(s) of radiation delivered to the body in order to killcancer cells

I High doses of radiation (energy/unit mass) can kill cellsand/or prevent them from growing and dividing

I true for cancer cells and normal cells

I Treatment works because the repair mechanisms for cancercells is less e�cient than for normal cells

IOE 610: LP II, Fall 2013 Examples of LP models: Radiation Therapy Page 24

Treatment delivery illustration

A linear accelerator and atreatment couch

A view of a collimator(beam-shaping device)


Radiation delivery — single beam direction

10

9

8

7

6

tumor

Relative Intensity of Dose Delivered


Radiation delivery — multiple beam directions

9

tumor 5

4

2

1

5

4

2

2

1

1

5 4

Relative Intensity of Dose Delivered


A schematic representation of a patient’s anatomy

151

151

T is the target areaC is the critical areaN is normal tissueS := T [ C [N

Treatment planning problem

I For a given tumor and given critical areas

I For a given set of possible “beamlet” origins and anglesI Determine the intensity of each beamlet such that:

I dosage over the tumor area will be at least a target level �L

I dosage over the critical area will be at most a target level �U


Building a model of the treatment planning problem — I

Divide up the treatment region into a 2-dimensional (or3-dimensional) grid of pixels

pixel (i,j)

i

j


Building a model of the treatment planning problem — II

Create the beamlet data for each of p = 1, . . . , n possiblebeamlets. Dp is the matrix of unit doses delivered by beam p.

0

0

0

0

0

0

0.9

0.9

1.0

0

0.8

0.9

0.9

0

0

0.8

0

0

0

0

0

0

0

1.0

1.0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

1.0

Dp

ij

= dose delivered to pixel (i , j) by beamlet p at unit intensity


Building a model of the treatment planning problem — III

Decision variables:

I wp

: intensity of “beamlet” p, p = 1, . . . , n

I Dij

: total dose delivered to pixel (i , j) for all pixels

I Relationship (stated as constraints):

Dij

=nX

p=1

Dp

ij

wp

for each beamlet (i , j)


Building a model of the treatment planning problem — III

minimizew ,D

X

(i ,j)2S

Dij

Dij

=nX

p=1

Dp

ij

wp

(i , j) 2 S

wp

� 0 p = 1, . . . , n

Dij

� �L

(i , j) 2 T

Dij

�U

(i , j) 2 C

I Unfortunately, this model is typically infeasible.

I Cannot deliver dose to the tumor without some harm tocritical area(s).


Possible practical formulation

Let (Target)i j

be the target prescribed dose to be delivered topixel (i , j)

minimizew ,D max

(i ,j)2S|D

i j

� (Target)i j

|

s.t. Di j

=P

n

p=1Dp

i j

wp

(i , j) 2 Sw � 0

This is the same as:

minimizew ,D,µ µ

s.t. Di j

=P

n

p=1Dp

i j

wp

(i , j) 2 S�µ D

i j

� (Target)i j

µ (i , j) 2 Sw � 0

(why?)


Problems with absolute values

(P1) minimizex

nX

j=1

cj

|xj

|

s.t. a0i

x � bi

, i 2 M1,

where cj

� 0 for all j = 1, . . . , nApproach 1

I |xj

| is a PWLC function: |xj

| = max(xj

, �xj

)I A non-negative multiple of a PWLC function is also a PWLC

functionI Sum of PWLC functions is also PWLC functionsI Equivalent re-formulation:

(P2) minimizex,z

nX

j=1

cj

zj

s.t. a0i

x � bi

i 2 M1,xj

zj

, j = 1, . . . , n�x

j

zj

, j = 1, . . . , nIOE 610: LP II, Fall 2013 Problems with absolute values Page 34

Problems with absolute values — continued

(P1) minimizex

nX

j=1

cj

|xj

|

s.t. a0i

x � bi

, i 2 M1,

where cj

� 0 for all j = 1, . . . , nApproach 2

I Introduce new variables x+j

, x�j

� 0, where xj

= x+j

� x�j

, foreach j

I The intention is to have x�j

= 0 when xj

> 0, and x+j

= 0when x

j

< 0I Re-formulation:

(P3) minimizex+,x�nX

j=1

cj

(x+j

+ x�j

)

s.t. a0i

x+ � a0i

x� � bi

i 2 M1,x+, x� � 0

I Are (P1) and (P3) equivalent?IOE 610: LP II, Fall 2013 Problems with absolute values Page 35

Treatment planning problem: Another possible formulation

minimizew ,D

X

(i ,j)2S

|Dij

� (Target)ij

|

s.t. Dij

=P

n

p=1Dp

ij

wp

(i , j) 2 S

wp

� 0 p = 1, . . . , n

This is equivalent to:

minimizew ,D,�

X

(i ,j)2S

�ij

s.t. Dij

=P

n

p=1Dp

ij

wp

(i , j) 2 S

��ij

Dij

� (Target)ij

�ij

(i , j) 2 S

wp

� 0 p = 1, . . . , n


A simple case study

151

151

(Target)ij

= 16, (i , j) 2 T(Target)

ij

= 0, (i , j) 2 C(Target)

ij

= 0, (i , j) 2 N

minimizew ,D,� 1 ·

X

(i ,j)2N

�ij

+ 100 ·X

(i ,j)2C

�ij

+ 30 ·X

(i ,j)2T

�ij

s.t. Dij

=P

n

p=1Dp

ij

wp

, (i , j) 2 S

��ij

Dij

� (Target)ij

�ij

, (i , j) 2 S

wp

� 0, p = 1, . . . , n


Optimal solution of the linear programming model

20 40 60 80 100 120 140 160 180 200

20

40

60

80

100

120

140

160

180

200


Algorithms and Software for solving LPs

I Algorithms:I Simplex method (“pivoting” method)I Interior-point method (IPM) (“barrier” method)

I Software codes (“solvers”):I CPLEX simplex (pivoting method)I CPLEX barrierI many other implementations

I Iteration Counts:I Number of pivots for simplex methodI Number of Newton steps for IPM


What a↵ects the running time of a solver on a probleminstance?

I Running time may be a↵ected by:I number of constraintsI number of variablesI type of algorithm (simplex or IPM)I software code (i.e., implementation of the algorithm)I properties of linear algebra systems involved

I density/patterns of nonzeroes of matrix systems to be solved

I other characteristics specific to problemI idiosyncratic influencesI pre-processing heuristics


Dimensional analysis of the modelSize of the underlying problem

151

151 In our formulation we used:

I number of beamlets = 564 (n)I number of pixels = 31, 397(⇡ ⇡ ⇤ 1002)

I |T | = 3, 859; |C| = 630; |N | = 26, 908I |S| = 31, 397


Dimensional analysis of the modelNumber of model variables


X

(i ,j)2N

�i j

+ 100 ·X

(i ,j)2C

�i j

+ 30 ·X

(i ,j)2T

�i j

s.t. Di j

=P

n

p=1Dp

i j

wp

(i , j) 2 S��

i j

Di j

� (Target)i j

�i j

(i , j) 2 Sw � 0

Decision Variables Number

wp

564Di j

31, 397�

i j

31, 397Total 63, 358


Dimensional analysis of the modelNumber of model constraints


X

(i ,j)2N

�i j

+ 100 ·X

(i ,j)2C

�i j

+ 30 ·X

(i ,j)2T

�i j

s.t. Di j

=P

n

p=1Dp

i j

wp

(i , j) 2 S��

i j

Di j

� (Target)i j

�i j

(i , j) 2 Sw � 0

Simple Variable Upper/Lower Bounds Number

w � 0 564Total 564

Other Constraints3 NumberDij

= 31, 397 D

ij

� (Target)ij

62, 794Total 94, 191

3We usually exclude simple variable upper/lower bounds when countingconstraints.


Dimensional analysis of the modelSummary

Variables Constraints*

63, 358 94, 191

*Excludes variable upper/lower bounds.


Some questions:

I What are the conceptual ideas behind the Simplex and Barriermethods?

I They perform reasonably well in practice on this problem, butare there theoretical guarantees of their performance quality?

I What issues arise in implementation of these algorithm?

I Can these algorithms be extended to tackle problems outsidethe LP family?


IOE 610/Math 660: Linear Programming IImepelman/teaching/IOE610/Handouts/610In… · IOE 610/Math...

Documents

Transcript of IOE 610/Math 660: Linear Programming IImepelman/teaching/IOE610/Handouts/610In… · IOE 610/Math...