Lp (2)

Linear Programming

Linear programming became important during World War II:→ used to solve logistics problems for the military.

Linear Programming (LP) was the first widely used form ofoptimization in the process industries and is still the dominantform today.

Linear Programming has been used to:

→ schedule production,→ choose feedstocks,→ determine new product feasibility,→ handle constraints for process controllers,

There is still a considerable amount of research taking place inthis area today.

M

Linear Programming

All Linear Programs can be written in the form:

⇒ note that all of the functions in this optimization problem are linear in the variables xi.

min + + + +

subject to:

+ + + +

+ + + +

+ + + +

xi

c x c x c x c x

a x a x a x a x b

a x a x a x a x b

a x a x a x a x b

n n

n n

n n

m m m mn n m

1 1 2 2 3 3

11 1 12 2 13 3 1 1

21 1 22 2 23 3 2 2

1 1 2 2 3 3

K

K

K

M M

K

!

!

!

0

0

,! !

"

x x

b

i l i

i

Linear Programming

In mathematical short-hand this problem can be re-written:

Note that:

1) the objective function is linear,2) the constraints are linear,

3) the variables are defined as non-negative,4) all elements of b are non-negative.

0b

xx

bx A

xcx

0

:subject to

min T

!

""

"

l

convention

convention

Linear Programming

A Simple Example:

A market gardener has a small plot of land she wishes to useto plant cabbages and tomatoes. From past experience sheknows that her garden will yield about 1.5 tons of cabbage peracre or 2 tons of tomatoes per acre. Her experience also tellsher that cabbages require 20 lbs of fertilizer per acre, whereastomatoes require 60 lbs of fertilizer per acre. She expects thather efforts will produce $600/ton of cabbages and $750/ton oftomatoes.

Unfortunately, the truck she uses to take her produce to marketis getting old, so she doesn't think that it will transportanymore than 3 tons of produce to market at harvest time. Shehas also found that she only has 60 lbs of fertilizer.

What combination of cabbages and tomatoes should she plantto maximize her profits?

Linear Programming

The optimization problem is:

In the general matrix LP form:

Linear Programming

We can solve this problem graphically:

We know that for well-posed LP problems, the solution lies ata vertex of the feasible region. So all we have to do is try all ofthe constraint intersection points.

c (a

cres

)

t (acres)

fertilizer

trucking

increasingprofit

feasibleregion

1 2

1

2

3

profit contours

Linear Programming

Check each vertex:

So our market gardener should plant 1.2 acres in cabbages and0.6 acres in tomatoes. She will then make $1980 in profit.

This graphical procedure is adequate when the optimizationproblem is simple. For optimization problems which includemore variables and constraints this approach becomesimpractical.

c t P(x)

0

2

0

6/5

0

0

1

3/5

0

1800

1500

1980

Linear Programming

Our example was well-posed, which means there was anunique optimum. It is possible to formulate LP problemswhich are ill-posed or degenerate. There are three main typesof degeneracy which must be avoided in Linear Program-ming:

1) unbounded solutions,

- not enough independent constraints.

2) no feasible region,

- too many or conflicting constraints.

3) non-unique solution,

- too many solutions,- dependence between profit function and a

few of the constraints.

Linear Programming

1) Unbounded Solution:

As we move up either the x1-axis or the constraint theobjective function improves without limit.

In industrial-scale problems with many variables andconstraints, this situation can arise when:

→ there are too few constraints.→ there is linear dependence among the constraints.

x 1

x2

improvingperformance

constraint

Linear Programming

2) No Feasible Region:

Any value of x violates some set of constraints.

This situation occurs most often in industrial-scale LPproblems when too many constraints are specified andsome of them conflict.

x 1

x2


Linear Programming

3) Non-Unique Solution:

Profit contour is parallel to constraint or:

x 1

x2


feasibleregion

optimum P(x) anywhere on this line segment.

iC Pxx

!"!

Linear Programming

Our market gardener example had the form:

We need a more systematic approach to solving theseproblems, particularly when there are many variables andconstraints.

→ SIMPLEX method (Dantzig).→ always move to a vertex which improves the value of

the objective function.

[ ]

[ ] . tomatoesacrescabbages acres :where

60

3

6020

21.5

:subject to

1500900- min

T!

"#

$%&

'("

#

$%&

'

x

x

xx

Linear Programming

SIMPLEX Algorithm

1) Convert problem to standard form with:• positive right-hand sides,• lower bounds of zero.

2) Introduce slack / surplus variables:• change all inequality constraints to equality

constraints.

3) Define an initial feasible basis:• choose a starting set of variable values which

satisfy all of the constraints.

4) Determine a new basis which improves objective function:• select a new vertex with a better value of P(x).

5) Transform the equations:• perform row reduction on the equation set.

6) Repeat Steps 4 & 5 until no more improvement in the objective function is possible.

Linear Programming

Market Gardener Revisited

The problem was written:

1) in standard form, the problem is:

[ ]

[ ] .tc where

60

3

6020

21.5

:subject to

1500900- min

T !

"

#$

%&'

()#

$

%&'

(

x

0x

x

xx

60 t 60 + c 20

3 t 2 + c 1.5

:subject to

t 1500 + c 900 tc,

max

!

!

trucking

fertilizer

Linear Programming


2) Introduce slack variables ( or convert all inequality constraints to equality constraints):

or in matrix form:0 s,st,c,

60 = s + t 60 + c 20

3 = s + t 2 + c 1.5

:subject to

t 1500 - c 900- tc,

min

21

2

1

!

[ ]

[ ] .sstc where

60

3

106020

0121.5

:subject to

001500900- min

T

21!

"

#$

%&'

(=#

$

%&'

(

x

0x

x

xx

trucking

fertilizer

Linear Programming


3) Define an initial basis and set up tableau:

• choose the slacks as the initial basic variables,• this is equivalent to starting at the origin.

the initial tableau is:

objective function value

c t s1 s2 b

s1s2

1.520.0

-900

2.060.0

-1500

1.00.0

0.0

0.01.0

0.0

3.060.0

0.0

basisvariables

objective function coefficients

Linear Programming


4) Determine the new basis:• examine the objective function coefficients and

choose a variable with a negative weight. (youwant to decrease the objective function becauseyou are minimizing. Usually we will choose themost negative weight, but this is not necessary).

• this variable will be brought into the basis.• divide each element of b by the corresponding

constraint coefficient of the new basic variable.• the variable which will be removed from the basis

is in the pivot row (given by the smallest positiveratio of bi/aij ).

c t s1 s2 b

s1s2

1.520.0

-900

2.060.0

-1500

1.00.0

0.0

0.01.0

0.0

3.060.0

0.0

most negative coefficient,bring "t" into the basis

b1/a12 = 3/2b2/a22 = 1

pivot

new basicvariable

new non-basicvariable

Linear Programming


5) Transform the constraint equations:• perform row reduction on the constraint equations

to make the pivot element 1 and all other elementsof the pivot column 0.

c t s1 s2 b

s1t

5/61/3

-400

0.01.0

0.0

1.00.0

0.0

-1/301/60

25.0

1.01.0

1500

i) new row #2 = row #2 / 60ii) new row #1 = row #1 - 2*new row #2iii) new row #3 = row #3 + 1500* new row #2

Linear Programming


6) Repeat Steps 4 & 5 until no more improvement is possible:

c t s1 s2 b

s1t

5/61/3

-400

0.01.0

0.0

1.00.0

0.0

-1/301/60

25.0

1.01.0

1500

c t s1 s2 b

ct

1.00.0

0.0

0.01.0

0.0

6/5-1/3

480

-1/253/100

9.0

6/53/5

1980

pivot

the new tableau is:

no further improvements are possible, since there are nomore negative coefficients in the bottom row of the tableau.

Linear Programming


The final tableau is:

optimal acres ofcabbages & tomatoes

c t s1 s2 b

ct

1.00.0

0.0

0.01.0

0.0

6/5-1/3

480

-1/253/100

9.0

6/53/5

1980

optimal basis contains bothcabbages and tomatoes. maximum

profit

shadow pricesReduced costDual variablesLagrange MultipliersKuhn-Tucker multipliers

Linear Programming


Recall that we could graph the market garden problem:

We can track how the SIMPLEX algorithm moved through thefeasible region. The SIMPLEX algorithm started at the origin.Then moved along the tomato axis (this was equivalent tointroducing tomatoes into the basis) to the fertilizer constraint.The algorithm then moved up the fertilizer constraint until itfound the intersection with the trucking constraint, which isthe optimal solution.

c (a

cres

)

t (acres)

fertilizer

trucking

increasingprofit

feasibleregion

1 2

1

2

3

profit contoursx*

start

SIMPLEX Algorithm (Matrix Form)

To develop the matrix form for the Linear Programmingproblem:

we introduce the slack variables and partition x, A and c asfollows:

Then, the Linear Programming problem becomes:

min

:

x

c x

Ax b

x 0

T

subject to

!

"

[ ]

x

x

x

A B N

c

c

c

=

=

=

B

N

B

N

! ! !

"

#

$$$

%

&

'''

! ! !

"

#

$$$

%

&

'''

M

basic

non-basic

min

:

x

c x c x

Bx Nx b

x x 0

+

,

T T

B B N N

B N

B N

subject to

+

=

!


Feasible values of the basic variables (xB) can be defined interms of the values for non-basic variables (xN):

The value of the objective function is given by:

or:

Then, the tableau we used to solve these problems can berepresented as:

[ ]x B b NxB N = -

-1

[ ]PB N N N

( )x c B b Nx c x = -T -1 T+

[ ]PB N B N

( )x c B b c c B N x = -T -1 T T -1+

I B-1 NxB

xBT xNT

0

b

B-1b

[ ]- -T T -1c c B NN B c B b

B

T -1


The Simplex Algorithm is:

1) form the B and N matrices. Calculate B-1.

2) calculate the shadow prices (reduced costs) of the non- basic variables (xN):

3) calculate B-1 N and B-1 b.

4) find the pivot element by performing the ratio test using the column corresponding to the most negative shadow price.

5) the pivot column corresponds to the new basic variable and the pivot row corresponds to the new non-basic variable. Modify the B and N matrices accordingly. Calculate B-1.

6) repeat steps 2 through 5 until there are no negative shadow prices remaining.

[ ]- -T T -1c c B NN B


This method is computationally inefficient because you mustcalculate a complete inverse of the matrix B at each iterationof the SIMPLEX algorithm. There are several variations onthe Revised SIMPLEX algorithm which attempt to minimizethe computations and memory requirements of the method.Examples of these can be found in:

ChvatalEdgar & Himmelblau (references in §7.7)FletcherGill, Murray and Wright

Duality & Linear Programming

Many of the traditional advanced topics in LP analysis andresearch are based on the so-called method of Lagrange. Thisapproach to constrained optimization:

. originated from the study of orbital motion.

. has proven extremely useful in the development of,• optimality conditions for constrained problems,• duality theory,• optimization algorithms,• sensitivity analysis.

Consider the general Linear programming problem:

This problem is often called the Primal problem to indicatethat it is defined in terms of the Primal variables (x).

minx

c x

Ax b

:

T

subject to

!


You form the Lagrangian of this optimization problem asfollows:

Notice that the Lagrangian is a scalar function of two sets ofvariables: the Primal variables x, and the Dual variables (orLagrange multipliers) λ. Up until now we have been callingthe Lagrange multipliers λ the shadow prices.

We can develop the Dual problem first be re-writing theLagrangian as:

which can be re-arranged to yield:

This Lagrangian looks very similar to the previous one exceptthat the Lagrange multipliers λ and problem variables x haveswitched places.

[ ]bAxxcx !TT - = ),( ""L

objective functionP(x)

constraintsg(x)

Lagrange multipliers

[ ]!! TTTT - = ),( bAxcxx "L

[ ]cAxbx !""" TTT - = ),(L


In fact this re-arranged form is the Lagrangian for themaximization problem:

This formulation is often called the Dual problem to indicatethat it is defined in terms of the Dual variables λ.

It is worth noting that in our market gardener example, if wecalculate the optimum value of the objective function usingthe Primal variables x*:

and using the Dual variables λ*:

we get the same result. There is a simple explanation for thisthat we will examine later.

max

T

T

0

cA

b

!

"

#

#

##

:subject to

1980 = 5

31500

5

6 900 = = )( *T

Primal

*!"

#$%

&+!"

#$%

&xcxP

( ) ( ) 1980 = 9604803 = = )( *T

Dual

* +!! bP


Besides being of theoretical interest, the Dual problemformulation can have practical advantages in terms of ease ofsolution. Consider that the Primal problem was formulated as:

As we saw earlier, problems with inequality constraints of thisform often present some difficulty in determining an initialfeasible starting point. They usually require a Phase 1 startingprocedure such as the Big ‘M’ method.

The Dual problem had the form:

Problems such as these are usually easy to start since λ =0 is afeasible point. Thus no Phase 1 starting procedure is required.

As a result you may want to consider solving the Dualproblem, when the origin is not a feasible starting point for thePrimal problem.

minx

c x

Ax b

:

T

subject to

!

max

T

T

0

cA

b

!

"

#

#

##

:subject to

Optimality Conditions & Linear Programming

Consider the linear programming problem:

• if equality constraints are present, they can beused to eliminate some of the elements of x,thereby reducing the dimensionality of theoptimization problem.

• rows of the coefficient matrix (M) are linearlyindependent.

In our market gardener example, the problem looked like:

minx

c x

M

Nx

b

b

subject to:

T

M

N

!

"#

$

%& '

!

"#

$

%&

active

inactive

c (a

cres

)

t (acres)

g1

feasibleregion

1 2

1

2

g2

g3

g4

x*

activeg1 - fertilizerg2 - truckingg3 - non-negativity tomatoesg4 - non-negativity cabbages

Form the Lagrangian:

At the optimum, we know that the Lagrange multipliers(shadow prices) for the inactive inequality constraints are zero(i.e. λN=0). Also, since at the optimum the active inequalityconstraints are exactly satisfied:

Then, notice that at the optimum:

At the optimum, we have seen that the shadow prices for theactive inequality constraints must all be non-negative (i.e. λM≥0). These multiplier values told us how much the optimumvalue of the objective function would increase if the associatedconstraint was moved into the feasible region, for aminimization problem.

Finally, the optimum (x*,λ*) must be a stationary point of theLagrangian:


]-[ - ]-[ - = ),( N

T

NM

T

M

TbNxbMxxcx !!!L

Mx b 0*! =

M

*T** = )( = ),(*

xcxx PL !

( ) ( )( ) ( ) TT*T***

TT*T*T****

=- =)( = ),(

=-=)( -)P( = ),(

0bMxxgx

0Mcxgxx xx

MM

MMM

L

L

!

!!!

l"

""


Thus, necessary and sufficient conditions for an optimum of aLinear Programming problem are:

1) the rows of the active set matrix (M) must be linearly independent,

2) the active set are exactly satisfied at the optimum point x*:

3) the Lagrange multipliers for the inequality constraints are:

this is sometimes expressed as:

4) the optimum point (x*,λ*) is a stationary point of the Lagrangian:

Mx b*=

M

T** = ),( 0xx

!L"

0

0

*

*

!

=

M

N

"

"

( ) [ ] ( ) [ ] 0bNxbMx +*T**T* =!!

NNMM""

This 2 variable optimization problem has an optimum at theintersection of the g1 and g2 constraints, and can be depictedas:


x1

x2

feasibleregion

profit contours

g1

g2

∇x g1

∇x g2

∇x P

Linear Programming & Sensitivity Analysis

Usually there is some uncertainty associated with the valuesused in an optimization problem. After we have solved anoptimization problem, we are often interested in knowing howthe optimal solution will change as specific problemparameters change. This is called by a variety of names,including:

• sensitivity analysis,• post-optimality analysis,• parametric programming.

In our Linear Programming problems we have made use ofpricing information, but we know such prices can fluctuate.Similarly the demand / availability information we have usedis also subject to some fluctuation. Consider that in ourmarket gardener problem, we might be interested indetermining how the optimal solution changes when:

• the price of vegetables changes,• the amount of available fertilizer changes,• we buy a new truck.

In the market gardener example, we had:

Changes in the pricing information (cT) affects the slope of theobjective function contours.

Changes to the right-hand sides of the inequality constraints(b), translates the constraints without affecting their slope.

Changes to the coefficient matrix of the constraints (A) affectsthe slopes of the corresponding constraints.

c (a

cres

)

t (acres)

fertilizer

trucking

increasingprofit

feasibleregion

1 2

1

2

3

profit contours



For the general Linear Programming problem:

we had the Lagrangian:

or in terms of the active and inactive inequality constraints:

Recall that at the optimum:

Notice that the first two sets of equations are linear in thevariables of interest (x*,λM*) and they can be solved to yield:

[ ]

0

0bMxx

0Mcxx

=

= - = ),L(

= - = ),L(

T**

TTT**

!

"

"

N

M

M

#

#

##

#

M

M

bMx

cM

=

)( =

1-*

1- T*!

minx

c x

Ax b

:

T

subject to

!

[ ]bAxxcxgxx !!TTT = )()( = ),( """ -PL

[ ] [ ]NNMM

L bNxbMxxcx !!!!TTT = ),( """


Then, as we discussed previously, it follows that:

We now have expressions for the complete solution of ourLinear Programming problem in terms of the problemparameters.

In this course we will only consider two types of variation inour nominal optimization problem including:

1) changes in the right-hand side of constraints (b). Such changes occur with variation in the supply / demand of materials, product quality requirements and so forth.2) pricing (c) changes. The economics used in optimization are rarely known exactly.

For the more difficult problem of determining the effects ofuncertainty in the coefficient matrix (A) on the optimizationresults see:

Gal, T., Postoptimal Analysis, Parametric Programming andRelated Topics, McGraw-Hill, 1979.

Forbes, J.F., and T.E. Marlin, Model Accuracy for EconomicOptimizing Controllers: The Bias Update Case,Ind. Eng. Chem. Res., 33, pp. 1919-29, 1994.

*TT*-1T*T* = )( = = = )(PMMMMM!! bbbMcxcx


For small (differential) changes in bM, which do not changethe active constraint set:

Note that the Lagrange multipliers are not a function of bM aslong as the active constraint set does not change.

For small (differential) changes in c, which do not change theactive constraint set:

( )[ ] ( )

[ ]

! !

! ! " " "

#

$

%%%

&

'

(((

! ! " " "

#

$

%%%

&

'

(((

b b

b b

b b

x b

x M b

M

0

M c

0

0

P( ) = =

= =

= =

* T T

* -1

-1

*

-1

l l

l

* *

M

shadow prices

inverse of constraintcoefficient matrix

[ ][ ]

!!!

"

#

$$$

%

&

'''

!!!

"

#

$$$

%

&

'''((

((

(('

0

M

0

cM

0bMx

xxcx

cc

cc

cc

1-T1-T

*

1*

T**T*

)(

=

)(

=

= =

)( = = )P(

)

M

Note that x* is not an explicit function of the vector c and willnot change so long as the active set remains fixed. If a changein c causes an element of λ to become negative, then x* willjump to a new vertex.

Also it is worth noting that the optimal values P(x*), x* and λ*

are all affected by changes in the elements of the constraintcoefficient matrix (M). This is beyond the scope of the courseand the interested student can find some detail in thereferences previously provided.

There is a simple geometrical interpretation for most of theseresults. Consider the 2 variable Linear Programming problem:

Note that the set of inequality constraints can be expressed:


min

:

( )

( )

x ,xc x c x

subject to

g

g

a x a x

a x a x

1 21 2 2

1

2

11 1 12 2

21 1 22 2

1 +

!

"

###

$

%

&&&

=

+

+

!

"

###

$

%

&&&

= '

x

x Ax b

M M

Ax

M

N

x x b

x

x

= ! ! !

"

#

$$$

%

&

'''

=

(

(

! ! !

"

#

$$$$

%

&

''''

)

g

g

1

2

M

Linear Programming

Summary

Linear Programs (LP) have the form:

the solution of a well-posed LP is uniquely determinedby the intersection of the active constraints.

most LPs are solved using the SIMPLEX method.

commercial computer codes used the RevisedSIMPLEX algorithm (more efficient).

optimization studies include:• solution to the nominal optimization problem,• a sensitivity study to determine how uncertainty

in the assumed problem parameter values affectthe optimum.

upper

T

0

:subject to

min

xx

bxA

xcx

!!

!

Interior Point Methods

Recall that the SIMPLEX method solved the LP problem by“walking” around the boundary of the feasible region. Thealgorithm moved from vertex to vertex on the boundary.

Interior Point methods move through the feasible regiontoward the optimum. This is accomplished by:

1) Converting the LP problem into an unconstrainedproblem using “Barrier Functions” for the constraintsat each new point (iterate),

2) Solving the optimality conditions for the resultingunconstrained problem,

3) Repeating until converged.

This is a very active research area and some excitingdevelopments have occurred in the last two decades.

Interior Point Methodsc

(acr

es)

t (acres)

fertilizer

trucking

increasingprofit

1 2

1

2

3

profit contours

Linear Programming Problems

1. Kirkman Brothers Ice Cream Parlours sell three differentflavours of Dairy Sweet ice milk: chocolate, vanilla, andbanana. Due to extremely hot weather and a high demand forits products, Kirkman has run short of its supply ofingredients: milk, sugar and cream. Hence, Kirkman will notbe able to fill all of the orders its has from its retail outlets (theice cream parlours). Due to these circumstances, Kirkman hasto make the best amounts of the three flavours given therestricted supply of the basic ingredients. The company willthen ration the ice milk to its retail outlets.

Kirkman has collected the following data on profitability ofthe various flavours and amounts of each ingredient requiredfor each flavour:

Kirkman has 180 gal. of milk, 150 lbs. of sugar and 60 gal. ofcream available.

milk(gal/gal)

0.450.500.40

Sugar(lbs/gal)

0.500.400.40

cream(gal/gal)

0.100.150.20

Profit($/gal)

$1.00$0.90$0.95

flavour

chocolatevanillabanana

Linear Programming Problems

Continued

a) What mix of ice cream flavours will maximize thecompany’s profit. Report your complete solution.

b) Using your solution determine the effects of changes inthe availability of milk, sugar and cream on the profit.

c) What is the most effective way to increase your profit?

Workshop Problems - Linear Programming

2. A jeweler makes rings, earrings, pins, and necklaces. She wishes towork no more than 40 hours per week. It takes her 2 hours to make aring, 2 hours to make a pair of earrings, 1 hour to make a pin and 4hours to make a necklace. She estimates that she can sell no morethan 10 rings, 10 pairs of earrings, 15 pins and 3 necklaces in asingle week. The jeweler charges $50 for a ring, $80 for a pair ofearrings, $25 for a pin, and $200 for a necklace. She would like todetermine how many rings, pairs of earrings, pins and necklaces sheshould make each week in order to produce the largest possible grossearnings.

a) What mix of jewelry will maximize the company’s profit.Report your complete solution.

b) Using your solution determine the effects of changes in theavailability of number of work hours per week, number ofhours to required to produce each type of jewelry and thedemand for each type of jewelry on the profit.

c) What is the most effective way to increase your profit?d) Comment on whether you think it was appropriate to solve

this problem with a conventional LP algorithm.Then x* is an inflection point. Otherwise, if the first higher-order,non-zero derivative is even:

Lp (2)

Technology

Transcript of Lp (2)