Multi-variable Unconstrained Optimization:

Unidirectional Search

Dr. Nasir M Mirza

Optimization TechniquesOptimization Techniques

Email: nasirmm@yahoo.com

In this Lecture …• In this lecture we will discuss two important methods

that deal with Multi-variable Unconstrained Optimization:

• Multi – Dimensional Plots and

• Uni-directional Search

Example 4.2The following function of two variables has distinct local minimum points and inflection point:

f = (x - 2)2 + (x - 2у2)2

Let us draw a mesh – grid graph in three dimensions using MATLAB.

% matlab function for 3D plot clear all [X,Y] = meshgrid(-2:.1:2);Z = ( X - 2.0).^2 + (X - 2.*Y.*Y).^2 ; mesh(X, Y, Z) %interpolatedaxis tight; hold onplot3(x,y,z,'.','MarkerSize',15)

% matlab function for 3D plot clear all [X,Y] = meshgrid(-2:.1:2);Z = ( X - 2.0).^2 + (X - 2.*Y.*Y).^2 ; surf(X,Y,Z,'FaceColor','red','EdgeColor','none')camlight left; lighting phong

SURFACE PLOT using MATLAB

Example 4.2 continued• Object function is: f = (x - 2)2+ (x - 2у2)2

⎟⎟⎠

⎞⎜⎜⎝

⎛

+−−+−

=

⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜

⎝

⎛

∂∂∂∂

=3

2

168444yxyyx

yfxf

vectorgradient

⎟⎟⎠

⎞⎜⎜⎝

⎛+−−−

=

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

∂∂

∂∂∂

∂∂∂

∂∂

= 2

2

22

2

2

2

488884

yxyy

yf

xyf

yxf

xf

H

32

22

168)2(8

444)2(2)2(2

yxyyxyyf

yxyxxxf

+−=−−=∂∂

−+−=−+−=∂∂

% matlab function for 3D plot clear all [X,Y] = meshgrid(-2:.05:5); Z = ( X - 2.0).^2 + (X - 2.*Y.*Y).^2 ; contour(X,Y,Z, 3000)

Next step is to draw a contour graph and identify points of interest.

(Here we see (2, -1)T and (2, 1)T as important points.

CONTOUR PLOT using MATLAB

% matlab function for 3D plot clear all [X,Y] = meshgrid(-2:.05:5); Z = ( X - 2.0).^2 + (X - 2.*Y.*Y).^2 ; surf(X,Y,Z,'FaceColor','red','EdgeColor','none')camlight left; lighting phonghold oncontour(X,Y,Z, 1000)

SURFACE PLOT using MATLAB

Example 4.2 continuedTwo types of contour plots showing possible maximum or minimum points

Example 4.2• Optimality condition (first order):

0168)2(8

0444)2(2)2(2

32

22

≡+−=−−=∂∂

≡−+−=−+−=∂∂

yxyyxyyf

yxyxxxf

Possible solutions (x, y) are following:

(1, 0) ; (2, -1); (2, 1)

Let us check with second optimality condition for each point:

For point (1, 0):

⎟⎟⎠

⎞⎜⎜⎝

⎛−

=80

04H

Principal minors are:

A1 = a11= 4, A2 = det(A)= -32

It is then inflection point and Function value = 2.

Example 4.2Let us check with second optimality condition for point (2, -1)

⎟⎟⎠

⎞⎜⎜⎝

⎛=

32884

HPrincipal minors are:

A1 = a11= 4, A2 = det(A)= 128 – 64 = 64

It is then minimum point and Function value = 0.

Let us check with second optimality condition for point (2, +1)

⎟⎟⎠

⎞⎜⎜⎝

⎛−

−=

32884

HPrincipal minors are:

A1 = a11= 4, A2 = det(A)= 128 – 64 = 64

It is then minimum point and Function value = 0.

Example 4.3• The following function of two variables has only one minimum point

but two distinct local maximum points and two inflection points:

f(x) = 25x2 – 12x4 – 6xy + 25y2 – 24x2y2 – 12y4.

% matlab function for 3D plot clear all [X,Y] = meshgrid(-1.5:.05:1.5); Z = 25.*X.^2-12.*X.^4-6.*X.*Y +25.*Y.^2 -24.*(X.^2).*(Y.^2) -12.*Y.^4. surf(X,Y,Z,'FaceColor','red','EdgeColor','none')camlight left; lighting phonghold oncontour(X,Y,Z, 1000)

Let us first draw 3D graph:

% matlab function for this graphclear all [X,Y] = meshgrid(-1.5:.1:1.5); Z = 25.*X.^2-12.*X.^4-6.*X.*Y +25.*Y.^2 -24.*(X.^2).*(Y.^2) - 12.*Y.^4. surf(X, Y, Z);colormap(jet);

Example 4.3

Example 4.3

% matlab function for 3D plot clear all [X,Y] = meshgrid(-1.5:.01:1.5); Z = 25.*X.^2-12.*X.^4-6.*X.*Y +25.*Y.^2 -24.*(X.^2).*(Y.^2) -12.*Y.^4. contour(X, Y, Z, 200);

Example 4.3• Object function is: f(x) = 25x2 – 12x4 – 6xy + 25y2 – 24x2y2 – 12y4.

⎟⎟⎠

⎞⎜⎜⎝

⎛

−−−−−−−−

=

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

∂∂

∂∂∂

∂∂∂

∂∂

=22

22

2

22

2

2

2

14448509669664814450

yxxyxyyx

yf

xyf

yxf

xf

H

32

23

4848506

4864850

yyxyxyf

xyyxxxf

−−+−=∂∂

−−−=∂∂

Example 4.3• Optimality condition (first order):

Possible solutions (x, y) are following:

(-0.764, 0.764) ; (-0.677, -0.677); (0, 0); (0.764, -0.764) ; (0.677, 0.677);

Let us check with second optimality condition for each point:

For point (-0.764, 0.764):

⎟⎟⎠

⎞⎜⎜⎝

⎛−

−=

62505062

H

Principal minors are:

A1 = a11= -62, A2 = det(A)= 1344

It is then maximum point and

Function value = 16.33

04848506

04864850

32

23

=−−+−=∂∂

=−−−=∂∂

yyxyxyf

xyyxxxf

Example 4.3Let us check with second optimality condition for point (-0.677, -0.677)

⎟⎟⎠

⎞⎜⎜⎝

⎛−−−−

=38505038

HPrincipal minors are:

A1 = a11= -38, A2 = det(A)= -1056

It is then inflection point and Function value = 10.08

Let us check with second optimality condition for point (0, 0)

⎟⎟⎠

⎞⎜⎜⎝

⎛−

−=

506650

HPrincipal minors are:

A1 = a11= 50, A2 = det(A)= 2464

It is then minimum point and Function value = 0.

Example 4.3Let us check with second optimality condition for point (+0.677, +0.677)

⎟⎟⎠

⎞⎜⎜⎝

⎛−−−−

=38505038

HPrincipal minors are:

A1 = a11= -38, A2 = det(A)= -1056

It is then inflection point and Function value = 10.08

Let us check with second optimality condition for point (0.764, -0.764)

⎟⎟⎠

⎞⎜⎜⎝

⎛−

−=

62505062

HPrincipal minors are:

A1 = a11= -62, A2 = det(A)= 1364

It is then maximum point and

Function value = 16.3

Example 4.3

The graph shows these points on contour plot as maximum, minimum and inflection points;

Maximum

Maximum

Inflection

Inflection

Minimum

Unidirectional Search Method• A unidirectional search is a one-dimensional search

performed by comparing function values only along a specific direction.

• Usually a unidirectional search is performed from a point x(t) in a specific direction s(t).

• Only points that lie on a line passing through x(t) in direction of s(t) are allowed.

• Any arbitrary point on that line can be expressed as

x(a) = x(t) + as(t).

• The parameter a is a scalar quantity and it is measure of distance of point x(a) from x(t).

Unidirectional Search Method

Once optimal a is found the corresponding point is found.

• The above equation is a vector equation in multi-dimensional space.

• Hence when a is known the x(a) can be estimated.

• Any positive or negative value of a will create x(a) on the line for search.

• When a is zero, the current point x(t) is obtained.

• When we want minimum point, we convert objective function in terms of single variable a by putting all xi as

x(a) = x(t) + as(t).

Example 3.2.1 Consider the objective function:

Minimize: f(x, y) = (x – 10)2 + (y – 10)2

• Step 1: First draw the contour plot of this function. The contour plot has lines where any two points on the line have the same function value.

• The optimal point is a minimum at (10, 10)T. The function value is zero at this point.

• Step 2: Let us say that a current point selected is x(t) = (2, 1)T. and the function value in a search direction s(t) = (2, 5)T from the current point. x(a) = x(t) + as(t) ; vector equation

x(a) = 2 + a2 ; y(a) = 1 + a5; scalar equations

Example 3.2.1 • Now we find the equation for the straight line passing

through (2, 1)T and direction (2, 5)T as:(x – 2)/2 = (y – 1)/5 ; y – 1 = (5/2)(x – 2).

• Now let us find a point on this line as x = 3 then y = 3.5 and we can estimate a from above equations as a = 0.5

• If the point on line is with x = 9 and y = 18.5, then a = 3.5

• The function value for (3, 3.5)T is 91.25 • And at point (9, 18.5)T the value is 73.25.• Then at a = (3.5 + 0.5)/2 = 2, then x = 6, y = 11 and

function value is 17. So, approx. opt. is (6, 11)T.

Example 3.3.1Consider the Himmelblau function: Minimize: f(x, y) = (x2 + y – 11)2 + (x + y2 –7)2

in the interval 0 < x < 5;

Step 1: This function is plotted as Figure. It appears from the plot that a minimum point is at (3, 2)T.

% matlab function for 3D plot clear all [X,Y] = meshgrid(0:.05:5); Z = ( X.*X + Y - 11.0).^2 + (X + Y.*Y - 7.0).^2 ; surf(X, Y, Z);colormap(jet);

% matlab function for contour plot clear all axis([0 5 0 5])hold on[X,Y] = meshgrid(-2:.05:5); Z = ( X.*X + Y - 11.0).^2 + (X + Y.*Y - 7.0).^2 ; contour(X, Y, Z, 200)

Example 3.3.1: contour linesA contour line is a collection of points having identical function values.

The contour plot for the object function is shown here.

The continual decrease in the function value of the successive contour lines as they approach a point means that the point is minimum point.

Example 3.3.1Step 1: We now choose an initial point x(0) = (1, 1)T

Size reduction parameter = (2, 2)T

Also choose e = 0.001

Step 2: We create a two dimensional hypercube (a square) around x(0) such that

X(1) = (0, 0)T , x(2) = (2, 0)T ,

X(3) = (0, 2)T , x(4) = (2, 2)T

Step 3: The function values at these points are:

f(0) = 106 ; f(1) = 170 ; f(2) = 74

f(3) = 90 ; f(4) = 26 The minimum is at (2, 2)T. Let us say this as x* =(2, 2)T

Example 3.3.1Step 4: Since x* is not x(0), we now set x(0) = (2, 2)T and got to step 2.

Step 2: We create a new square around this x(0) with Δ = 1 such that

X(1) = (1, 1)T , x(2) = (3, 1)T ,

X(3) = (1, 3)T , x(4) = (3, 3)T

Step 3: The function values at these points are:

f(0) = 26 ; f(1) = 106 ; f(2) = 10

f(3) = 58 ; f(4) = 26 The minimum is at (3, 1)T. The function value is 10.

Let us say this as x* =(3, 1)T

Example 3.3.1Step 4: Since x* is not x(0), we now set x(0) = (3, 1)T and go to step 2.

Step 2: We create a new square around this x(0) with Δ = 1 such that

X(1) = (2, 0)T , x(2) = (4, 0)T ,

X(3) = (2, 2)T , x(4) = (4, 2)T

Step 3: The function values at these points are:

f(0) = 10 ; f(1) = 74 ; f(2) = 34

f(3) = 26 ; f(4) = 50 The minimum is at (3, 1)T where function value is 10. It is same as x(0).

Let us say this as x* =(3, 1)T

Example 3.3.1Step 4: Since x* is same as x(0), we now set x(0) = (3, 1)T and go to step 2 after reducing step size = 0.5

Step 2: We create a new square around this x(0)=(3, 1)T with Δ = 0.5 such that

X(1) = (2.5, 0.5)T , x(2) = (3.5, 0.5)T

,

X(3) = (2.5, 1.5)T , x(4) = (3.5, 1.5)T

Step 3: The function values at these points are computed and minimum is at x(4) = (3.5, 1.5)T

And function value is 9.125.

The minimum is at (3.5, 1.5)T

Let us say this as x* =(3.5, 1.5)T

Example 3.3.1Step 4: Since x* is not same as x(0), we now set x(0) = (3.5, 1.5)T

and go to step 2 with step size = 0.5

Step 2: We create a new square around this x(0)=(3, 1)T with Δ = 0.5 such that

X(1) = (3, 1)T , x(2) = (4, 1)T ,

X(3) = (3, 2)T , x(4) = (4, 2)T

Step 3: The function values at these points are

f(0) =9.125; f(1) = 10 ;

f(2) = 40; f(3) = 0; f(4) = 50

the minimum is f(3) = 0.0

The minimum is at (3, 2)T

This is the best minimum so far as f=0.

Example 3.3.1It is important to see that although we have found a minimum point but the algorithm does not terminate at this step. We may continue.

It is clear that the convergence depends upon the initial cube size and location and the chosen size reduction parameter.

Small size may lead to premature convergence and large may never lead to convergence.

The minimum is at (3, 2)T

This is the best minimum so far as f=0.

Example: Suppose f(x, y) = 2xy + 2x – x2 – 2y2

Using the steepest ascent method to find the next point if we are moving from point (-1, 1).

yxyfxy

xf 42222 −=

∂∂

−+=∂∂

⎥⎦

⎤⎢⎣

⎡−

=⎥⎦

⎤⎢⎣

⎡−−

−−+=

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

−∂∂

−∂∂

=∇−6

6)1(4)1(2

)1(22)1(2

)1,1(

)1,1(),1,1(At

yfxf

f

)6161()(Let hh,-fhg −+=

Next step is to find h that maximize g(h)

2.0036072yields0)('Setting

180727...)6161()(222)(

2

22

=⇒=−=

−+−==−+=

−−+=

hhhg

hhhh,-fhgyxxxyxf

If h = 0.2 maximizes g(h), then x = -1+6(0.2) = 0.2 and y = 1-6(0.2) = -0.2 would maximize f(x, y).

So moving along the direction of gradient from point (-1, 1), we would reach the optimum point (which is our next point) at (0.2, -0.2).