Lec 20 Multivar Cauchy Method

8/9/2019 Lec 20 Multivar Cauchy Method

1/26

Multi-variable Unconstrained Optimization:

Cauchy method and Newton Method

Dr. Nasir M Mirza

Optimization Techniques Optimization Techniques

Email: [email protected]


2/26

In this Lecture …

• In this lecture we will discuss two important methods

that deal with Multi-variable Unconstrained

Optimization:

• Cauchy’s Steepest Ascent Method

• Newton's Method


3/26

Cauchy’s Steepest Ascent Method

• The search direction used in Cauchy’s method is the

negative of the gradient at any particular point x*.

• Since this direction gives maximum descent in

function values, it is also known as the steepest

descent method.

• At every iteration the derivative is computed at

current point and a unidirectional search is performed

in negative to this derivative direction to findmaximum.

• The maximum becomes the current point and search

is continued from this point.


4/26


5/26

Example: Suppose f ( x, y) = 2 xy + 2 x – x2 – 2 y2

Using the steepest ascent method to find the next point if we aremoving from point (-1, 1).

y x y

f x y

x

f 42222 −=

∂∂−+=

∂∂

⎥⎦⎤⎢

⎣⎡−

=⎥⎦⎤⎢

⎣⎡

−−−−+=

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

−∂

∂

−∂

∂

=∇−6

6)1(4)1(2

)1(22)1(2

)1,1(

)1,1(

),1,1(At

y

f x

f

f

)6161()(Let hh,- f h g −+=

Next step is to find h that maximize g (h)


6/26

2.0036072

yields0)('Setting

180727...)6161()(

222)(

2

22

=⇒=−

=

−+−==−+=

−−+=

hh

h g

hhhh,- f h g

y x x xy x f

If h = 0.2 maximizes g (h), then x = -1+6(0.2) = 0.2 and y = 1-6(0.2) = -0.2 would maximize f ( x, y).

So moving along the direction of gradient from point (-1,1), we would reach the optimum point (which is our nextpoint) at (0.2, -0.2).


7/26

EXERCISE 3.4.2

Consider the Himmelblau

function:

Minimize:f(x, y) = (x2 + y – 11)2 + (x + y2

– 7)2

•Step 1: In order to ensure

proper convergence, a large value

of M (= 100) is usually chosen.

The choice of M also depends on the available time and computing

resource. Let us choose M = 100, an initial point x(0) = (0, 0)T, and

termination parameters ε1 = ε2 = 10-3

. We also set k = 0.


8/26

EXERCISE 3.4.2

3D Graph


9/26

Example

% Matlab program to

draw contour of function

[X,Y] = meshgrid(0:.1:5);

Z = (X.*X + Y -11.).^2. +(

X + Y.*Y - 7.).^2.contour(X, Y, Z, 150);

colormap(jet); Minimum point

Contour graph:


10/26

EXERCISE 3.4.2

h

h x f h x f

x

f ii

i 2

)()( −−+=

∂

∂

Step 2:

The derivative at x(0) = (0, 0)T is

first calculated

and found to be (-14, -22)T, whichis identical to the exact derivative

at that point (Figure 3.12).

Step 3:The magnitude of the derivative vector is not small and k =a < M =

100. Thus, we do not terminate; rather we proceed to Step 4.


11/26

EXERCISE 3.4.2

• Step 4: At this step, we need to perform a line search from x(0)

in the direction - f(x(0)) such that the function value is minimum.

• Along that direction, any point can be expressed by fixing a value orthe parameter a(0) in the equation:

x = x(0) - a(0) f(x(0)) = (14a(0), 22a(0)) T .

1.0

yields)1,0(0)('Setting

)714484)(14968(2)1122196)(22392(2)(

)714484()1122196()2214()(

)7()11()(

22

2

2222

2222

=

=

−+++−++=′

−++−+==

−++−+=

a

region for a g

aaaaaaa g

aaaaaa, f a g

y x y x x f


12/26

EXERCISE 3.4.2

• Thus, the minimum point along the search direction isx(1) == (1.788, 2.810)T.

• Step 5: Since, x(1) and x(0) are quite different, we donot terminate; rather we move back to Step 2. Thiscompletes one iteration of Cauchy's method. The totalnumber of function evaluations required in this iterationis equal to 30.

• Step 2: The derivative vector at this point, computednumerically, is (-30.7, 18.8) T .


13/26

EXERCISE 3.4.2

• Step 3: This magnitude of the derivative vector is not

smaller than ε1. Thus, we continue with Step 4.

• Step 4: Another unidirectional search along (30.70, -

18.80) T from the point x(1) = (1.788, 2.810)T using

the golden section search finds the new point:

• x(2) = (3.00, 1.99)T with a function value equal to 0.018.


14/26

Newton’s Method

• This method uses second – order derivatives to createsearch directions.

• This allows faster convergence to the minimum point.• Consider the first three terms in Taylor’s series

expansion of a multivariable function, it can be shown

that the first order optimality condition will be satisfiedif following search direction is used:

[ ] )()( )(1

)(2)( k k k x f x f s ∇∇−= −


15/26

Newton's Method

One-dimensionalOptimization

Multi-dimensionalOptimization

At theoptimal

Newton's

Method

0)( =∇ x f 0)(' =i x f

)(1

1 iiii f xHxx ∇−= −

+

)(")('

1

i

iii

x f x f x x −=+

Hi is the Hessian matrix (or

matrix of 2nd

partialderivatives) of f evaluated

at xi.)(')(" 1 ii x f x f

−


16/26

Newton's Method

• Converge quadratic fashion.

• May diverge if the starting point is not close

enough to the optimum point.

• Costly to evaluate H-1.

)(11 iiii f xHxx ∇−= −

+


17/26


18/26

EXERCISE 3.4.3

3D Graph


19/26

Example on Newton Method

% Matlab program todraw contour of function

[X,Y] = meshgrid(0:.1:5);

Z = (X.*X + Y -11.).^2. +(

X + Y.*Y - 7.).^2.

contour(X, Y, Z, 150);colormap(jet);

Minimum point

Contour graph:


20/26


( ) ( )

( ) ( )( ) ( )

[ ] .22000.42 and2214),0,0(At

621624

2442412

1422222272112

14424472114

711),(

T

2

2

3222

2322

2222

⎥⎦

⎤⎢⎣

⎡

−

−=−−=∇

⎥⎦

⎤⎢⎣

⎡

++−+

+−+=

−++−−=−++−+=∂

∂

−+−+=−++−+=

∂

∂

−++−+=

H

H

f

y x y x

y x y x

y y xy y x y x y y x y

f

y x xy x y x y x x x

f

y x y x y x f


21/26


( ) ( )

[ ]

⎟⎟ ⎠

⎞⎜⎜⎝

⎛

−

−=

⎟⎟ ⎠

⎞⎜⎜⎝

⎛

−

−⎟⎟ ⎠

⎞⎜⎜⎝

⎛

−

−−⎟⎟

⎠

⎞⎜⎜⎝

⎛ =

∇−=

⎥⎦⎤⎢

⎣⎡ −

−=−−=∇

−++−+=

−

a

a

a f a x x

f

y x y x y x f

333.0

22

14

420

022

9240

0)0()1(

.22000.42 and2214),0,0(At

711),(

1

T

2222

H

H


22/26


• Performing a unidirectional search along thisdirection, we obtain value of a = -3.349.

• Since this quantity is negative, the functionvalue does not reduce in the given searchdirection.

• Instead, function value reduces in the oppositedirection.

• This shows that search direction in Newtonmethod may not always be descent.

• When this happens we restart with a new point.


23/26


Now again select a new initial point x(0) = (2, 1)T,

and function value at this point is f(x(0)) = 52.

Step 2: The derivative at this point is calculated as (-57, -24)T.

Step 3: Since the termination criteria are not met we go to the step 4.Step 4: At this point the Hessian is given as:

( ) ( )

( ) ( )

( ) ( )

[ ] .510

1010 and2457),1,2(At

32224

2442412

1422222272112

14424472114

711),(

T

2

2

3222

2322

2222

⎥⎦⎤⎢

⎣⎡=−−=∇

⎥⎦

⎤⎢⎣

⎡

++−+

+−+=

−++−−=−++−+=∂

∂

−+−+=−++−+=∂∂

−++−+=

H

H

f

y x y x

y x y x

y y xy y x y x y y x y

f

y x xy x y x y x x x f

y x y x y x f


24/26


( ) ( )

[ ]

⎟⎟

⎠

⎞⎜⎜

⎝

⎛

+

−=

⎟⎟

⎠

⎞⎜⎜

⎝

⎛

−

−⎟⎟

⎠

⎞⎜⎜

⎝

⎛

−

−

−−⎟⎟

⎠

⎞⎜⎜

⎝

⎛ =

∇−=

⎥⎦⎤⎢

⎣⎡=−−=∇

−++−+=

−

a

a

a f a x x

f

y x y x y x f

6.61

9.0224

57

1010

105

501

2)0()1(

.510

1010 and2457),0,0(At

711),(

1

T

2222

H

H


25/26


• A unidirectional search along this directionreveals that the minimum occurs at a = 0.34

• Since this quantity is positive, we accept this.• The new point is then (1.694, 3.244) and

function value is 44.335 which is smaller than

the previous point value of 52.• It means we are going in right direction.

• When we continue for next two iterations we

reach at point (3.021, 2.003)T with f = 0.0178;• Hence answer is point (3, 2)T for this case when

we go further.


26/26

EXERCISE 3.4.3

• This method is effective for initial points close to the

optimum.

• This demands some knowledge of the optimum point.

• Computation of the Hessian matrix and its inverse is

computationally expensive.

Lec 20 Multivar Cauchy Method

Documents

Transcript of Lec 20 Multivar Cauchy Method