Crash Course in Mathematics Constrained...

Crash Course in Mathematics Constrained Optimisation

Crash Course in Mathematics

Constrained Optimisation

Serçin �ahin

Y�ld�z Technical University

9 October 2012


Equality Constraints

Formally we write the problem as follows:

maxx1,x2f (x1, x2) s.t. g(x1, x2) = 0

f (x1, x2) is called the objective function or maximand,

The x1, x2 are called choice variables,

The function g(x1, x2) is called the constraint and

The set of all x1, x2 that satisfy the constraint are sometimes

called the constraint set or the feasible set.



One way to solve this problem is by substitution.

Suppose that g(x1, x2) = 0 can be written to isolate x2 on one

side as:

x2 = g̃(x1)

We substitute it into the objective function:

maxx1f (x1, g̃(x1))

The usual �rst-order conditions required that we set the total

derivative,∂f

∂xequal to zero and solve for the optimal x∗1 .



Lagrange's Method

Let us reconsider our original problem:

maxx1,x2f (x1, x2) s.t. g(x1, x2) = 0

Let's form a new function, called Lagrangian, which has three

variables instead of two: namely x1, x2, λ:

L(x1, x2, λ) ≡ f (x1, x2)− λg(x1, x2)First-order conditions give the critical points of L:



Lagrange's Method

The General Form of the Lagrange's Method:

Suppose we have a function of n variables and we face mconstraints, where m < n. Our problem is:

For x = (x1, ..., xn) and λ = (λ1, ..., λn) , we obtain the

function of n +m variables:

The �rst-order conditions again require that all partial

derivatives of L be equal to zero at the optimum (x∗, λ∗):



Geometric Interpretation

We can represent the objective function geometrically by its

level sets

L(y0) ≡ {(x1, x2) | f (x1, x2) = y0}At any point along any level set of the function, total

di�erential must be zero:

Then the slope of the level set through (x1, x2) will be




We can think of the constraint function , as a kind of level set.

It is the set of all (x1, x2) such that

g(x1, x2) = 0

Again, for any (x1, x2) satisfying the constraint, the following

relation must hold:

Then the slope of the constraint at the point (x1, x2) is:




Simple rearrangement of �rst-order conditions for a critical

point of the Lagrangian function gives

Suppose λ∗ 6= 0.

Dividing the �rst of the equations by the second gives



Second-Order Conditions

The bordered Hessian of the Lagrangian function involves the

second-order partials of L bordered by the �rst-order partials

of the constraint equation and a zero.

Let's call the determinant of the bordered Hessian D.

If (x∗1 , x∗2 , λ

∗) solves the �rst-order conditions, and if D > 0

when evaluated at (x∗1 , x∗2 , λ

∗), then (x∗1 , x∗2 ) is a local

maximum of f (x1, x2) subject to the constraint g(x1, x2) = 0.

If (x∗1 , x∗2 , λ

∗) solves the �rst-order conditions, and if D < 0

when evaluated at (x∗1 , x∗2 , λ

∗), then (x∗1 , x∗2 ) is a local

minimum of f (x1, x2) subject to the constraint g(x1, x2) = 0.




In the multivariable, multiconsraint case, the bordered Hessian

is again formed by bordering the matrix of second-order

partials of L by all the �rst-order partials of the constraints

and enough zeros to form a partials of L by all the �rst-order

partials of the constraints and enough zeros to form a

symmetric matrix.

Its principal minors are the determinants of submatrices

obtained by moving down the principal diagonal.




The n −m principal minors of interest here are those

beginning with the (2m + 1) st and ending with the

(n +m)-th,i.e., the determinant of H.




Su�cient Conditions for Local Optima with Equality

Constraints

Let the objective function be f (x) and the m < n constraints

be g j(x) = 0, j = 1, ...m.Given the Lagrangian function, let (x∗, λ∗) solve the �rst-orderconditions of the Lagrange's method. Then,

1 x∗ is a local maximum of f (x) subject to the constraints if the

(n −m) principal minors alternate in sign beginning with

positive Dm+1 > 0,Dm+2 < 0, ..., when evaluated at (x∗, λ∗).2 x

∗ is a local minimum of f (x) subject to the constraints if the

(n −m) principal minors are all negative

Dm+1 < 0,Dm+2 < 0, ..., when evaluated at (x∗, λ∗).


Inequality Constraints

Let's begin with the simplest possible problem: maximising a

function of one variable subject to a non-negativity constraint

on the choice variable.

maxx f (x) subject to x ≥ 0.



All together, we have identi�ed three conditions thatcharacterise the solution to the simple maximisation problemwith non-negativity constraints.

Condition 1: f ′(x∗) ≤ 0

Condition 2: x∗ [f (x∗)]Condition 3: x∗ ≥ 0



Kuhn-Tucker Conditions

Let us now consider optimisation subject to general inequality

constraints


Optimality Theorems

Consider the maximisation problem,

where x is a vector of choice variables, and a = (a1, ..., al) is avector of parameters that may enter the objective function, the

constraints, or both.

Suppose that for each a ∈ A there is at least one solution,

x(a) to the maximisation problem stated above.

Then, for that vector of parameters, a, the maximised value of

the objective function is f (x(a), a). This de�nes a new

function, V (a), called the value function.


Optimality Theorems

The Envelope Theorem

Consider the maximisation problem when there is just one

constraint and suppose the objective function, f , and the

constraint function, g are continuously di�erentiable in (x, a).Let L(x, a, λ) be the problem's associated Lagrangian function

and let (x(a), λ(a)) solve the Kuhn-Tucker conditions.

�nally, let V (a) be the problem's associated value function.

Then, the Envelope theorem states that for every a ∈ U,

where the right-hand side denotes the partial derivative of the

Lagrangian function with respect to the parameter ajevaluated at the point (x(a), λ(a)).

Crash Course in Mathematics Constrained...

Documents

Transcript of Crash Course in Mathematics Constrained...