Post on 01-Feb-2016
description
Chapter 6 Method of Successive Quadratic ProgrammingProfessor Shi-Shang Jang
National Tsing-Hua University
Chemical Engineering Department
6-1 Quadratic Programming Problems
Problem:
0
..
'2
1)(min
:formmatrix compact In
0
..
11
11111
222
2222
11211221112211
x
bAxts
Qxxcxxf
x
bxaxa
bxaxats
xhxxhxh
xxhxxhxhxcxcxcMin
MNMNM
NN
NNNNN
NNNN
Classical Constrained Regression Problem
Where β is an m×1 vector of regression coefficients to be estimated, e is an n×1 vector of error variables, Y is an n×1 observations on the dependent variable and X is an n×m matrix of observations on the independent variables. It is clear that the classical regression problem is a quadratic programming problem.
exYts
Iee
..
'min
Example- A numerical problem
211..
42
2406min
formmatrix compact
0,
2 ..
2226)(
2
1
2
121
2
1
21
21
2221
211
x
xts
x
xxx
x
xxf
xx
xxts
xxxxxxf
Example- A numerical problem
2
11
42
24
06
b
A
Q
c
Example-A scientist’s model A scientist has observed a certain quantity Q
as a function of t. She has a good reason to believe that there is a physical law relating t and Q that takes the form:
Q(t)=asint+bcost+c
She wants to have the “best” possible idea of the value of the coefficients a,b and c using the results of her n experiments:
Example-A scientist’s model (t1,q1; t2,q2;…, tn,qn). Taking into account
that her experiments are not perfect and also perhaps that the model is not rigorous enough, she does not expect to find a perfect fit. So defines an error term:
Ei=qi-q(ti)
Her idea is to minimize:
n
iieZ
1
2min
Example-A scientist’s modelWhat if a=3;b=2;c=5 with 15 experimental data?
0cossin
0cossin
0cossin..
),,,,,(min
2222
1111
1
21
nnnn
n
iin
ectbtaq
ectbtaq
ectbtaqts
eeecbaf
Classical Least Square Fitting Problem: Given a set of data (x1,y1; x2,y2;…,
xn,yn). Find a set of parameters ={a, b, c,…} such that:
ei2=yi-(axi+bxi
2+..) And iei
2 is minimized The solution of the unconstrained case: =(XTX)-1XTY=quasi_inverse(X)Y
Example: Curve Fitting>> x=linspace(0,5);>> for i=1:100y(i)=3*x(i)^2+2*x(i)+5;y(i)=y(i)+randn(1,1);xx(i,1)=x(i)*x(i);xx(i,2)=x(i);xx(i,3)=1;end>> theta=inv(xx'*xx)*xx'*y'
theta =
2.9983 2.1215 4.5839
>> for i=1:100yy(i)=theta(1)*x(i)^2+theta(2)*x(i)+theta(3);end>> plot(x,yy)
Example: Curve Fitting-Continued
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50
10
20
30
40
50
60
70
80
90
100
Program QUADPROG QUADPROG Quadratic programming. X=QUADPROG(H,f,A,b) solves the quadratic programming problem: min 0.5*x'*H*x + f'*x subject to: A*x <= b x X=QUADPROG(H,f,A,b,Aeq,beq) solves the problem above while
additionally satisfying the equality constraints Aeq*x = beq. X=QUADPROG(H,f,A,b,Aeq,beq,LB,UB) defines a set of lower and upper bounds on the design variables, X, so that the solution is in the range LB <= X <= UB. Use empty matrices for LB and UB if no bounds exist. Set LB(i) = -Inf if X(i) is unbounded below; set UB(i) = Inf if X(i) is unbounded above.
Curve Fitting Using Quadprogx=linspace(0,5);for i=1:100 y(i)=3*x(i)^2+2*x(i)+5; y(i)=y(i)+randn(1); xx(i,1)=x(i)*x(i);xx(i,2)=x(i);xx(i,3)=1;enda=eye(100);a1=zeros(1,100);H=[a1; a1; a1; a];f=zeros(103,1);HH=[f f f H];Aeq=[xx eye(100)];beq=y';A=[];b=[];X=QUADPROG(HH,f,A,b,Aeq,beq) ;XX=X';XX(1)XX(2)XX(3)
The Solution Method-Complementary Pivot Problem K-T Condition for a QP:
0
0
0
0
u
ux
x
bAx
vAuQQxc TT
The Solution Method-Complementary Pivot Problem Find vectors w and z such that
0
0
0
zw
z
w
qMzw
T
The Solution Method-Complementary Pivot Problem The solution of the complementary pivot
problem is equivalent to find a K-T point of the original QP problem and hence the solution is sufficiently found since the QP is a convex objected (quadratic function) and qualified constrained (linear constraints).
6-2 The Successive QP Approach Consider a quadratic function:
The objective is to find d=(x-x0) such that the following problem can be minimized
002
00000 2
1; xxxfxxxxxfxfxxq T
0
0..
2
1min 2
dxgxg
dxhxhts
dxfddxf
Ttj
tj
Ttk
tk
tTTt
Example
0211
021..
244/25
4/258/3
2
1
4
47
4
23min
:found becan QP ingcorrespond The
12
01)(
02)(..
6)(min
0
21
21
212
121
d
dts
ddd
x
xxxg
xxxhts
xxxxxf
T
Example
Algorithm Step 1: Formulate the QP problem Step 2: Solve the QP problem, and set
x(t+1)=x(t)+*d Step 3: Check Convergency
Remarks Original SQP is subjected to the following
two problems:(1) The second derivative of the objective function is generally difficult to obtain.
(2) There is no guarantee that the second derivative of any function at any point is positive definite. In case the hessian is not positive definite, then the QP is failed.
6-3 Quadratic Approximation of the Lagrangian Function
0
0..
,,2
1min
:by edapproximat becan
0)(
0)(..
)(min
)()()()(
2
dxgxg
dxhxhts
dvuxLddxf
xg
xhts
xf
xugxvhxfxL
T
T
xTT
Algorithm-The Variable Metric Method Given initial estimates x0,u0,v0, and a symmetric
positive definitive matrix H0
Step 1: Solve the problem:
Step 2: Select the step size α along d(t), and set x(t+1)=x(t)+ α d(t)
0
0..
2
1min
dxgxg
dxhxhts
dHddxf
Ttt
Ttt
tTTt
Algorithm-The Variable Metric Method (VMCON) Step 3: Check convergence Step 4: Update H(t) using the gradient difference BSF update formulation:
wz
ww
zHz
HzzHHH
Finally
zHyw
then
yzzHz
zHzzHzyz
Define
vuxLvuxLy
xxz
T
T
tT
tTttt
t
TtT
tT
tTT
tttx
tttx
tt
1
111
1
1
otherwise 0.8
2.0 if 1
,,,,
Line Search on the SQP direction Given d from QP, instead of using the original
objective function, it is more useful to implement the following penalty function:
tj
tj
tj
tj
tk
tk
tk
tk
jjkk
J
jjj
K
kkk
J
jj
K
kk
uu
vv
uv
xgxhxf
xgxhRxfRxP
1
1
11
11
2
1,max
2
1,max
,iterationt subsequent allfor and
,
iterationfirst for the where
,0min()(
,0min()(),(
Numerical Example
T
T
TTT
d
d
dts
ddd
IH
ghfx
xxxg
xxxhts
xxxxxf
2 ,4
:givessolution The
0211
021..
10
01
2
1
4
47
4
23min
choosingby form the takerun willfirst The
1 1;21;4
47
4
23;12
01)(
02)(..
6)(min
0
1
0
21
21
212
121
MATLAB Program - VMCON% DEFINE GLOBAL VARIABLESglobal muglobal sigmaglobal vvglobal uuglobal ddglobal X0%% INITIALIZE THE PROBLEMH=eye(2);X0=[2;1];uu=0;vv=0;dx=100;mu=0;sigma=0;iter=0;%% THE WHILE LOOP%while abs(dx)>0.000001 iter=iter+1; % % FIND THE GRADIENT df % [df,dlgn]=sqp_Lagrangian(X0); % % DEFINE Aq,beq such that Aq*d=beq % A, b such that Ad<b
MATLAB Program – VMCON-Continued A=[-1 -1];b=(X0(1)+X0(2)-1); Aeq=[X0(2) X0(1)]; beq=(X0(1)*X0(2)-2);% % SOLVE THE SUB-QUADRATIC PROBLEM FOR d % [ss,FVAL,EXITFLAG,OUTPUT,LAMBDA]=QUADPROG(H,df,A,b,Aeq,beq); dd=ss'; % % GET THE LARAGNGE=MULTIPLIERS % uu=LAMBDA.ineqlin; vv=LAMBDA.eqlin; % % PREPARE THE LINE SEARCH % mu=max(abs(vv),0.5*(mu+abs(vv))); sigma=max(abs(uu),0.5*(sigma+abs(uu))); a0=0.1; % % LINE SEARCH ALONG dd WITH THE PENALTY FUNCTION AS THE OBJ % alopt = FMINSEARCH('sqp_penalty',a0);
MATLAB Program – VMCON-Continued% % GET THE NEW POINT % X=X0+alopt*dd'; % % PREPARE TO UPDATE THE APPROXIMATE HESSIAN %z=X-X0; [df,dlgn_1]=sqp_Lagrangian(X0); [df,dlgn_2]=sqp_Lagrangian(X); y=dlgn_2-dlgn_1; zz=z'*H*z; zzp=0.2*zz; zy=z'*y; if(zy>zzp) theta=1; else theta=0.8*z'*H*z/(zz-zy); end w=theta*y+(1-theta)*H*z; % % UPDATE THE APPROXIMATE HESSIAN % H=H-(H*z*z'*H)/zz+w*w'/(z'*w);
MATLAB Program – VMCON-Continued
% % CALCULATE THE OBJ AND EQUALITY CONSTRAINT AT THE NEW POINT % hh=X(1)*X(2)-2; ddx=X-X0; obj=sqp_obj(X); % % UPDATE THE NEW POINT % X0=X; % % CHECK THE CONVERGENCY % dx=sqrt(ddx(1)^2+ddx(2)^2);end%% OUTPUT THE RESULT%iterXobjhh
MATLAB Program Using Fmincon> help fmincon
FMINCON Finds a constrained minimum of a function of several variables.
FMINCON solves problems of the form: min F(X) subject to: A*X <= B, Aeq*X = Beq (linear
constraints) X C(X) <= 0, Ceq(X) = 0 (nonlinear
constraints) LB <= X <= UB
MATLAB Program Using Fmincon-Continued
X=FMINCON(FUN,X0,A,B,Aeq,Beq,LB,UB,NONLCON) subjects the minimization to the
constraints defined in NONLCON. The function NONLCON accepts X and returns
the vectors C and Ceq, representing the nonlinear inequalities and equalities
respectively. FMINCON minimizes FUN such that C(X)<=0 and Ceq(X)=0.
MATLAB Program Using Fmincon-ExampleX0=[2;1];A=[-1 -1];B=1;Aeq=[];Beq=[];LB=[];UB=[];X=FMINCON('sqp',X0,A,B,Aeq,Beq,LB,UB,'sqp_nlcon')
function [c,ceq]=sqp_nlcon(x)c=[];ceq=x(1)*x(2)-2;