Convex Optimization

CMU-10725Penalty Methods

Barnabás Póczos & Ryan Tibshirani

2

Outline

� Penalty functions

3

Books to Read

David G. Luenberger, Yinyu Ye: Linear and Nonlinear Programming

Boyd and Vandenberghe: Convex Optimization

4

Penalty Methods

5

Penalty Methods

Penalty program: replace (P) with the unconstrained problem:

Penalty term: high cost for violation of the constraints

6

Inequality Constraints

A useful penalty function in this case is:

7

Penalty Methods

David G. Luenberger, Yinyu Ye: Linear and Nonlinear Programming

8

Inequality and Equality Constraints

Inequality and Equality constraints:

Rewrite them as:

9

Penalty Method

Penalty parameter:

Penalty program:

Penalty Lemma:

10

Proof of Penalty Lemma (1)

11

Proof of Penalty Lemma (2)

12

Proof of Penalty Lemma (3)

13

Proof of Penalty Lemma (4)

14

Convergence of Penalty Method

Theorem: [Penalty convergence]

15

Proof of Penalty Convergence

16

Penalty functions

Often used penalty functions

Polynomial penalty:

Linear penalty:

Quadratic penalty:

17

Inequality and Equality constraints

Problem (P)

Definition [Penalty function]

Example [Penalty function]

18

Derivative of the penalty function

Penalty program:

Penalty function:

Assumptions:

Derivatives:

19

Derivative of the penalty function

Difficulties: max is not differentiable

This is not perfectly correct, because

Solution:

Example:

20

KKT in Penalty methods

Penalty program:

Penalty function:

Derivatives:

1st order condition in local minimum:

21

KKT and Penalty method multipliers

Penalty program:

Problem (P)

Theorem: Under some mild conditions

KKT multipliers: