1

OR IIOR IIGSLM 52800GSLM 52800

2

OutlineOutline

classical optimization – unconstrained optimization dimensions of optimization

feasible direction

3

Classical Optimization ResultsClassical Optimization Results Unconstrained Optimization Unconstrained Optimization

different dimensions of optimization conditions nature of conditions

necessary conditions (必要條件 ): satisfied by any minimum (and possibly by some non-minimum points)

sufficient conditions (充分條件 ): if satisfied by a point, implying that the point is a minimum (though some minima may not satisfy the conditions)

order of conditions first-order conditions: in terms of the first derivatives of f & gj

second-order conditions: in terms of the second derivatives of f & gj

general assumptions: f, g, gj C1 (i.e., once continuously differentiable) or C2 (i.e., twice continuously differentiable) as required by the conditions

4

Feasible Direction Feasible Direction

S n: the feasible region

x S: a feasible point

a feasible direction d of x: if there exists > 0 such that x+d S for 0 < <

5

Two Key Concepts Two Key Concepts for Classical Results for Classical Results

f: the direction of steepest accent

gradient of f at x0 being orthogonal to the tangent of the contour f(x) = c at x0

6

The Direction of Steepest Accent The Direction of Steepest Accent ff contours of f(x1, x2) = ff: direction of steepest accent in some sense, increment of unit

move depending on the angle with f f within within 9090 of of ff: increasing: increasing

closer to 0closer to 0: increasing more: increasing more

beyond beyond 9090 of of ff: decreasing: decreasing closer to 180closer to 180: decreasing more: decreasing more

above results generally true for above results generally true for any any ff

x2

x1

2 21 2x x

ff((xx11, , xx22) =) =

ff((xx1010, , xx2020) = ) = cc

d on the tangent plane at xd on the tangent plane at x00

ff(x(x00++d) d) cc for small for small

roughly speaking, for roughly speaking, for ff(x(x00) = ) = cc, , ff(x(x00++d) = d) = cc for for

small small when d is on the tangent plane at x when d is on the tangent plane at x00

7

Gradient of Gradient of ff at x at x00 Being Orthogonal to Being Orthogonal to

the Tangent of the Contour the Tangent of the Contour ff(x) = (x) = cc at x at x00

2 21 2x x

8

First-Order Necessary Condition First-Order Necessary Condition (FONC)(FONC)

ff CC11 on S and x on S and x** a local minimum of a local minimum of ff then for any feasible direction d at xthen for any feasible direction d at x**, , TTff(x(x**)d )d

0 0 increasing of increasing of ff at any feasible direction at any feasible direction

ff((xx) = ) = xx22 for 2 for 2 xx 5 5 ff((xx, , yy) = ) = xx22 + + yy22 for 0 for 0 xx, , yy 2 2

ff((xx, , yy) = ) = xx22 + + yy22 for for xx 3, 3, yy 3 3

9

FONC for Unconstrained NLPFONC for Unconstrained NLP

ff CC11 on S & x on S & x** an interior local minimum an interior local minimum (i.e., without touching any boundary) (i.e., without touching any boundary) TTff(x(x**) = 0) = 0

10

FONC Not SufficientFONC Not Sufficient

Example 3.2.2: f(x, y) = -(x2 + y2) for 0 x, y Tf((0, 0))d = 0 for all feasible direction d

(0, 0): a maximum point

Example 3.2.3: f(x) = x3

f(0) = 0

x = 0 a stationary point

11

Feasible Region with Feasible Region with Non-negativity ConstraintsNon-negativity Constraints

Example 3.2.4. (Example 10.8 of JB) Find candidates of the minimum points by the FONC. min f(x) =

subject to x1 0, x2 0, x2 0

* ** *( ) ( )

0, if 0; 0, if 0.j jj j

f fx x

x x

x x

2 2 21 2 3 1 2 1 3 13 2 2 2x x x x x x x x

* **( ) ( )

0, 0jj j

f fx

x x

x xor, equivalently

12

Second-Order ConditionsSecond-Order Conditions

another form of Taylor’s Theorem f(x) = f(x*)+Tf(x*)(x-x*)

+0.5(x- x*)TH(x*)(x - x*)+ ,

where being small, dominated by other terms

if Tf(x*)(x-x*) = 0, f(x) f(x*) (x- x*)TH(x*)(x - x*) 0

13

Second-Order Necessary ConditionSecond-Order Necessary Condition

ff CC22 on S on S

if xif x** is a local minimum of is a local minimum of ff, then for any , then for any feasible direction d feasible direction d nn at x at x**, ,

(i).(i). TTff(x(x**)d )d 0, and 0, and

(ii). if (ii). if TTff(x(x**)d = 0, then d)d = 0, then dTTH(xH(x**)d )d 0 0

14

Example 3.3.1(a)Example 3.3.1(a)

SONC satisfied

ff((xx) = ) = xx22 for 2 for 2 xx 5 5 ff((xx, , yy) = ) = xx22 + + yy22 for 0 for 0 xx, , yy 2 2

ff((xx, , yy) = ) = xx22 + + yy22 for for xx 3, 3, yy 3 3

15

Example 3.3.1(b)Example 3.3.1(b)

SONC: more discriminative than FONCSONC: more discriminative than FONC

ff((xx, , yy) = -() = -(xx22 + + yy22) for 0 ) for 0 xx, , y y in Example in Example 3.2.2 3.2.2

(0, 0), a maximum point, failing the SONC(0, 0), a maximum point, failing the SONC

16

SONC for Unconstrained NLPSONC for Unconstrained NLP

ff CC22 in S in S

xx** an interior local minimum of an interior local minimum of ff, then, then

(i).(i). TTff(x(x**) = 0, and ) = 0, and

(ii). for (ii). for allall d, d d, dTTH(xH(x**)d )d 0 0  (ii) H(xH(x**) being positive semi-definite ) being positive semi-definite

convex convex f f satisfying (ii) (and actually more)satisfying (ii) (and actually more)

17

Example 3.3.2Example 3.3.2

identity candidates of minimum points for the f(x) =

Tf(x*) =

x = (1, -1) or (-1, -1)

H(x) =

(1, -1) satisfying SONC but not (-1, -1)

3 21 2 1 23 2x x x x

21 1 2(3 3 ,2 2)x x x

16 0

0 2

x

18

SONC Not SufficientSONC Not Sufficient

ff((xx, , yy) = -() = -(xx44 + + yy44))

TTff((0, 0))d = 0 for all d((0, 0))d = 0 for all d

(0, 0) a maximum(0, 0) a maximum

19

SOSC for Unconstrained NLPSOSC for Unconstrained NLP

ff CC22 on S on S nn and and xx** an interior point an interior point

if if (i). (i). TTff(x(x**) = 0, and ) = 0, and

(ii). H(x(ii). H(x**) is positive definite) is positive definite

xx** a strict local minimum of a strict local minimum of ff

20

SOSC Not NecessarySOSC Not Necessary

Example 3.3.4.

x = 0 a minimum of f(x) = x4

SOSC not satisfied

21

Example 3.3.5Example 3.3.5

In Example 3.2.4, is (1, 1, 1) a minimum?

.

6 > 0;

positive definite, i.e., SOSC satisfied

6 2 2

( ) 2 2 0

2 0 2

H x

6 212 ( 2)( 2) 8;

2 2

6 2 2

2 2 0 (6)(2)(2) ( 2)(2)( 2) ( 2)( 2)(2) 8

2 0 2

2 2 21 2 3 1 2 3 1 2 1 3 1( , , ) 3 2 2 2f x x x x x x x x x x x

22

Effect of ConvexityEffect of Convexity

If for all y in the neighborhood of x* S, Tf(x*)(y-x*) 0

convexity of f implies f(y) f(x*) + Tf(x*)(y-x*) f(x*)

x* a local min of f in the neighborhood of x*

x* a global minimum of f

23

Effect of ConvexityEffect of Convexity

f C2 convex H positive semi-definite everywhere

Taylor's Theorem, when Tf(x*)(x-x*) = 0, f(x) = f(x*) + Tf(x*)(x-x*)

+ (x- x*)TH(x* + (1-)x)(x - x*)

= f(x*) + (x- x*)TH(x* + (1-)x)(x - x*)

f(x*)

x* a local min a global min

24

Effect of ConvexityEffect of Convexity

facts of convex functions  (i). a local min = a global min (ii). H(x) positive semi-definite everywhere (iii). strictly convex function, H(x) positive definite

everywhere

implications for f C2 convex function, the FONC Tf(x*) = 0 is

sufficient for x* to be a global minimum if f strictly convex, x* the unique global min