Optimization III: Constrained .Basic De nitions Feasible point and feasible set A feasible point

download Optimization III: Constrained .Basic De nitions Feasible point and feasible set A feasible point

of 33

  • date post

    09-Jan-2019
  • Category

    Documents

  • view

    215
  • download

    0

Embed Size (px)

Transcript of Optimization III: Constrained .Basic De nitions Feasible point and feasible set A feasible point

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Optimization III:Constrained Optimization

CS 205A:Mathematical Methods for Robotics, Vision, and Graphics

Doug James (and Justin Solomon)

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 1 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Announcements

I HW6 due today

I HW7 out

I HW8 (last homework) out next Thursday

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 2 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Constrained Problems

minimize f (~x)

such that g(~x) = ~0

h(~x) ~0

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 3 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Really Difficult!

Simultaneously:I Minimizing fI Finding roots of gI Finding feasible pointsof h

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 4 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Implicit Projection

Implicit surface: g(~x) = 0

Example: Closest point on surface

minimize~x ~x ~x02such that g(~x) = 0

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 5 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Implicit Projection

Implicit surface: g(~x) = 0

Example: Closest point on surface

minimize~x ~x ~x02such that g(~x) = 0

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 5 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Nonnegative Least-Squares

minimize~x A~x~b22such that ~x ~0

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 6 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Manufacturing

I m materials

I si units of material i in stock

I n products

I pj profit for product j

I Product j uses cij units of material i

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 7 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Manufacturing

Linear programming problem:

maximize~x

j pjxjsuch that xj 0 j

j cijxj si i

Maximize profits where you make a positive

amount of each product and use limited

material.

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 8 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Bundle Adjustment

min~yj ,Pi

ij Pi~yj ~xij22s.t. Pi orthogonal i

Applications:

I Bundler

I Building Rome in a Day

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 9 / 28

https://www.cs.cornell.edu/~snavely/bundlerhttp://grail.cs.washington.edu/projects/rome

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Constrained Problems

minimize f (~x)

such that g(~x) = ~0

h(~x) ~0

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 10 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Basic Definitions

Feasible point and feasible set

A feasible point is any point ~x satisfying g(~x) = ~0 andh(~x) ~0. The feasible set is the set of all points ~xsatisfying these constraints.

Critical point of constrained optimization

A critical point is one satisfying the constraints thatalso is a local maximum, minimum, or saddle point off within the feasible set.

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 11 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Basic Definitions

Feasible point and feasible set

A feasible point is any point ~x satisfying g(~x) = ~0 andh(~x) ~0. The feasible set is the set of all points ~xsatisfying these constraints.

Critical point of constrained optimization

A critical point is one satisfying the constraints thatalso is a local maximum, minimum, or saddle point off within the feasible set.

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 11 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Differential Optimality

Without h:

(~x,~) f (~x) ~ g(~x)Lagrange Multipliers

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 12 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Inequality Constraints at ~x

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 13 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Inequality Constraints at ~x

Two cases:

I Active: hi(~x) = 0Optimum might change if constraint is

removed

I Inactive: hi(~x) > 0Removing constraint does not change ~x

locally

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 14 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Idea

Remove inactive constraints and make active

constraints equality constraints.

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 15 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Lagrange Multipliers

(~x,~, ~) f(~x) ~ g(~x) ~ h(~x)No longer a critical point! But if we ignore that:

~0 = f(~x)i

igi(~x)j

jhj(~x)

jhj(~x) = 0

Zero out inactive constraints!

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 16 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Lagrange Multipliers

(~x,~, ~) f(~x) ~ g(~x) ~ h(~x)No longer a critical point! But if we ignore that:

~0 = f(~x)i

igi(~x)j

jhj(~x)

jhj(~x) = 0

Zero out inactive constraints!

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 16 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Inequality Direction

So far: Have not distinguished betweenhj(~x) 0 and hj(~x) 0

I Direction to decrease f : f (~x)I Direction to decrease hj: hj(~x)

f (~x) hj(~x) 0

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 17 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Inequality Direction

So far: Have not distinguished betweenhj(~x) 0 and hj(~x) 0

I Direction to decrease f : f (~x)I Direction to decrease hj: hj(~x)

f (~x) hj(~x) 0

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 17 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Inequality Direction

So far: Have not distinguished betweenhj(~x) 0 and hj(~x) 0

I Direction to decrease f : f (~x)I Direction to decrease hj: hj(~x)

f (~x) hj(~x) 0

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 17 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Dual Feasibility

j 0CS 205A: Mathematical Methods Optimization III: Constrained Optimization 18 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

KKT Conditions

Theorem (Karush-Kuhn-Tucker (KKT) conditions)

~x Rn is a critical point when there exist ~ Rm and~ Rp such that:

I ~0 = f(~x)

i igi(~x)

j jhj(~x)(stationarity)

I g(~x) = ~0 and h(~x) ~0 (primal feasibility)I jhj(~x

) = 0 for all j (complementary slackness)

I j 0 for all j (dual feasibility)

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 19 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

KKT Example from Book

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 20 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

KKT Example from Book

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 21 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Physical Illustration of KKT

Example: Minimal gravitational-potential-energyposition ~x = (x1, x2)

T of a particle attached toinextensible rod (of length `), and above a hardsurface.

minimize~x x2 (Minimize gravitational potential energy)such that ~x ~c2 ` = 0 (rod of length ` attached at ~c)

x2 0 (height 0)

Physical interpretation of f , g, h, and ?Physical interpretation of stationarity, primal feasibility,complementary slackness and dual feasibility?

CS 205A: Mathematical Methods Optimization III: Constrained Optimization 22 / 28

Announce Constrained Problems Motivation Optimality Algorithms Convex Optimization

Sequential Quadratic Programming(SQP)